posted under category: Software Quality on December 21, 2012 by Nathan
This year (2012, for posterity sake) my project at work took on an Agile methodology. Why? 2011 was a productive year, we made a lot of people happy with our software, but we had about three giant software releases for the whole year. These changes being so far between meant I could never know if what I was doing was going to work for everyone, and it also meant that, while the software was better tested, when there were bugs, they would have to work around them for months until we had the fix out.
Accelerating our release cycle to every two weeks meant that even if something was broken, it wouldn't stay broken for long. It also meant we had to be on top of our tasks a lot more, with a stronger focus on what we are doing today and what we will be doing tomorrow.
How did we do it? We initially decided on twice-weekly scrum meetings, just to say what we did and what we were doing next. This wasn't enough for the project manager, so we bumped it up to 4 times per week (the 5th day we had a regularly scheduled customer meeting). We began the year by listing all of our customers and all of their existing and upcoming tasks, including our own nit-list (like upgrade the database version, wash the dishes, support the new manufacturing system, etc). This is our backlog. We worked with our customers to set a priority on each item, then planned a sprint strategy, taking tasks with the highest priority and mixing them with the most pressing deadlines. Throughout the year we tracked new business and requests into an issues log, promoted items from our backlog into workable sprint items, and we got a lot of things done that way.
We did learn some lessons though. First, daily scrum meetings are great, but when our customer gets involved (and he loves to be involved), he tends to ask questions mid-update, slowing the scrum to a crawl, turning what should be a 5 minute meeting into a 45 minute meeting. We should really cut that out. Also we have the need for a system that will take a customer feedback and track it through an official change request to a prioritized backlog item, to an in-work task, to a bullet point on our release notes. There are things that almost do this, but usually require something custom for the customer feedback part at the very least, and so few of them have been 'blessed' for use within my company. Also we have some release systems that do not play well with Agile. One system that notifies the help desk about outages and work in the area takes an hour to fill out each time we do a production release. Another system we may be forced to use requires a gated software version validation process that will take a day's worth of paperwork for each release.
The worst downside though is that it sucks my energy. I have to be on task a lot more, which has meant less blogging and less tweeting. That's sad for me.
In summary, twenty-five software releases on our two-week sprint cycle, less stress on the development, more pressure on the project manager, and happier customers. All-in-all, it has been a successful experiment that I think we will continue with.
posted under category: Software Quality on March 13, 2012 by Nathan
A discussion erupted recently, on one of the various discussion groups i subscribe to, about deprecated features that are not really deprecated but more like, not encouraged when thinking about future applications moving forward in development time.
It's hard to explain in a definition, so a case in point is needed.
The Application.cfm file is a ColdFusion construct as old as I can remember. It's not going away, but it is essentially feature complete.
More recently, ColdFusion has given us Application.cfc. The capabilities that gives us are far above and beyond what a had previously, and developers are encouraged to upgrade their applications or at least use Application.cfc in their next projects.
In this case, Application.cfm has been socially deprecated.
Here's another one: Fusebox. It's not a language feature so much as a framework, but if your application uses Fusebox, your program is socially deprecated. Now my friend John is bringing it back, but for the past 5 years, if you have started a new app with Fusebox, you used a socially deprecated framework and your "cool kids" card has been revoked.
using socially deprecated features and frameworks puts your application at risk for falling into technical debt. It's a slow roll down a long hill that will show in bugs that are never fixed, areas of the application that everyone avoids because they are afraid of causing errors, and eventually, lack of knowledgeable developers which will leave you stranded.
The way to avoid social deprecation is to keep engaged with the developer community. Read the blogs, engage in the discussion groups and visit the user groups.
Can you think of anything else that's socially deprecated? I can think of a bunch of things. CFUpdate. Prototype.js. Old stylized code like capitalizing tag and attribute names. Php. Internet Explorer. CFPod. The list goes on and on.
posted under category: Software Quality on September 9, 2011 by Nathan
Last July I gave my presentation, Holistic Program Quality and Technical Debt to the Denver CFUG, for my friend John Blayter. This is cool and stuff, but the truly impressive thing is that the audio and video is consistent all the way through, and the awful jokes were even relatively funny.
If you haven't seen it, this is the definitive version. This is the one to sit through.
Watch Holistic Program Quality and Technical Debt
Thanks, John, for getting it up there, even though I may have insulted your programming around the 14:30 mark. Yeah, I apologized, then pointed out exactly where I threw you under the bus. That just happened.
Update: You can hear my awesome kids screaming their heads off around 42:00. Fantastic.
posted under category: Software Quality on July 1, 2011 by Nathan
This month it looks like I'm presenting Holistic Program Quality and Technical Debt (HPQaTD? Can someone pronounce this for me?), twice! If you haven't seen it yet, you have two chances in July.
Tuesday, July 12, for the Denver CFUG and my friend John Blayter. Read about it and RSVP at The Denver CFUG's Meetup site. It's at 5:30 PDT / MST, or 6:30 MDT in Colorado. John ran the CFUG in Phoenix for a few years, and actually was my boss at Interactive Sites until he moved to Colorado.
Thursday, July 14, for the Central Georgia CFUG and my friend Tim Cunningham. I have not met the user group manager Matt Abbott, but I warn you not to misspell his name. You can see the event and RSVP at The CGCFUG's Adobe Groups site. This one is at 3:30 PDT / MST, or 6:30 ET, Tim hooked me on to this as a trade in services for his presentation on Git at the June Phoenix CFUG.
Both of these are remote presentations. If you want the address, you should follow me on twitter, where I tend to post only the most important subjects, and links to cool speeches.
Thanks for the interest in my talk!
posted under category: Software Quality on May 14, 2011 by Nathan
A number of peope asked to have my presentation slides from my talk on Holistic Program Quality and Technical Debt, so here they are.
And here are a few sample slides:
Thanks to everyone who came, especially those I pestered into coming. I really do appreciate it.
posted under category: Software Quality on April 12, 2011 by Nathan
We left the easy list of things to do that will help you pay down your technical debt, but left off with refactoring. If you wanted a nice, vague discussion, you could read Wikipedia on Code Refactoring, but you won't find anything you can take home. It would be easier reading the definition, which just quotes the only important part of the Wikipedia article:
Code refactoring is the process of changing a computer program's source code without modifying its external functional behavior in order to improve some of the nonfunctional attributes of the software
So how do you refactor? There are three places that I see it.
First, if you obeyed the golden rule of technical debt, you are keeping track of your loans. Refactoring in this sense is nothing more than finding those areas in your application and working in what needs to be done. It's easy, which is why you need to follow that golden rule. The huge advantage here is that you can put estimates on it and schedule it into your project. This is really more like a budgeted technical debt payment, but it can still fit the refactoring label.
Another path to refactoring is to overcome your knowledge gap. If you wrote a program, you probably did it as well as you could — minus the gaps you know about, but the gaps you don't know about are where your best improvements lie. That's where true refactoring comes in.
I was talking about this the other day with subversively fixing technical debt: reading a book is the best time to come up with new ideas. I guess it's obvious what books I read. The design patterns book was especially great for this. Reading along to tech books sparks all kinds of new ideas. More often, they are unrelated to the content I am reading, not always.
Once you learn what your gap is, and you visualize how to fix it, there is only a matter of time before you get to see it through. This process is why I believe that if you want to have refactoring improvements in your code, (1) you have to be learning, and (2) they almost always come organically.
These organic enhancements like to start deep down in my thinking. Often I read something that produces a small shift in what a good program should look like. Eventually that idea turns into something I can do within my application, like introduce a new kind of object in just the right spot, break a form down into reusable parts, or develop a useful independent API that makes everything feel better. One day I eventually work up a physical understanding of the code that needs to change. I introduce the modification in a safe development branch, then I see if it fits. It usually does, but if not, I roll it back, no damage done.
The last kind of refactoring is incidental refactoring. I do this all the time.
Often while working one programming task, I find ways to enhance things that we never knew needed enhancements. These tend to be little things. Whenever I am in one area of an application I take ownership in, I look around to see if there is anything else I can improve.
Incidental refactoring is the best. It keeps a system fresh, it fixes those little bugs and near-bugs, it is the driving force to cleaning up ugly code, and it makes customers happy.
Incidental refactoring does not happen all the time. There is a single way that I know to produce it, and that way is ownership. You don't plant a tree in front of someone else's house unless they ask you to. You probably wouldn't rewrite a funny loop in a program you have just been introduced to unless you were tasked with it (although I have met a few brazen enough). When you have ownership of an application, you check in capitalizations and misspelled variables. Ownership means caring because the application's customers are your customers, the application's success is your success.
Refactoring is an ongoing process. You work at it and you figure something else out, you think about it more and something new comes to mind. Refactoring is a process, not a task. It's a lifestyle not a goal.
posted under category: Software Quality on April 11, 2011 by Nathan
I realize a lot of my technical debt blog writing has been all talk. That's fine, but boring. What actual, practical thing can you do now - today - that pays down your technical debt?
(By the way, I'm talking about this stuff at CF.Objective() 2011, May 12-14 in Minneapolis. It's a month away so right now is the time to book your tickets. Come meet me and see my presentation Holistic Program Quality and Technical Debt!)
Going deeper into code readability, you can take it beyond tabs and line breaks. Commenting and naming are as important. Comment strategies can vary per team, but decide on a preference. Components, functions, files and variables should all have obvious names, not too short, always consistent. Again, standardize it with the team. You can alleviate almost all of the stress of working with a complex system simply by making the code readable.
I hate to say this. It's 2011, I shouldn't have to. If you don't have source control, you have done yourself a ginormous disservice. If you are working on your own, you will love what a source control system will do for you. If you work on a team without it, you are handicapping every aspect of the team's ability. If your boss tells you "no," tell your boss "yes."
Beyond this lies the territory of refactoring, which encompasses everything larger than changing variable names and line breaks. I'm talking about practical refactoring next time. Stay tuned.
posted under category: Software Quality on April 9, 2011 by Nathan
I have been talking about my friend, who was in a technical-debt-pickle when he called me. Something else he mentioned over the phone was that the owners of his giant application were not keen on paying for anything but features. It's not surprising.
In an agile project, work is done in sprints 1-3 weeks in length. Teams set goals for features they can accomplish in that time. When application customers hear about a refactoring sprint, it goes over like a lead balloon. "You want to work for 2 weeks and not add features?" Like Oracle and open source, they just don't get it.
So what do you do when you can't get a budget for fixing the last decade of technical debt build-up?
I gave him my best advice, and some of my points leaned toward subverting the wishes of uninformed management and customers. Without putting your job in jeopardy, I think this is something you have to do sometimes. But how do you do it? How do you pay down some of the technical debt principal when managers say no?
First, you need to have the technical debt conversation with the customers & owners. Help them realize that by allowing this debt to live on, they are perpetrating higher interest payments in IT man hours. No sane management group would deny you the time you need, however, I know that a lot of companies are run by the most insane people.
Shot down or passed over from there, you have to pad estimates. You have been estimating high anyways because of those technical debt interest payments where it takes half of your time trying not to break other modules, finding the right files to copy and paste, or trying to remember how to use the terrible API left by some former employee. Pad higher and take that extra time to fix some of the bigger problems.
I find that, when I get myself fully in to a program, where I can feel it breathe, I start to dream about it. It's is a good thing. All my best system thinking tends to happen while I'm reading a book. Offline, I discover the pattern and identify new objects that will help me jump that next hurdle. At work the next day, I merely have to type what I envisioned.
Refactoring doesn't take a lot of time, it takes better thinking. A little padding and some daydreams will get you there.
You may be able to add this padding to each release cycle, agile or not. Call it unit testing. Call it the software release time, call it the cycle preparation time, call it quality insurance. No reason to lie about it being there. Also, no reason to share the gritty details of what's being done.
With limited refactoring time, it's ok if a refactor project is stalled halfway through. When I leave one and pick it up again, I find that I have newer, better ideas.
Going back to the people you report to, I would point out my review of my friend Terrence Ryan's book Driving Technical Change, which really seems to fit the bill here.
The final way to subvert the authority for the betterment of all, and goes along with the golden rule of technical debt (keep it visible!), is to bring it up, often. It's easy to ignore a dog waiting patiently by the door, but when it keeps yapping, you'll want to let it out. It's easy to say no to some engineers who want to run an expensive project, but if they just won't shut up about it, it becomes easier to say yes. Once, my team begged the CEO of our small software company to rewrite an app. Then we begged some more. Then we begged some more. We pointed out problems, we started to shout it from the cube-top, and eventually, he gave in. It works.
I don't think this list is exhaustive by any means. Does anyone want to share their experience or ideas?
posted under category: Software Quality on April 2, 2011 by Nathan
So I was telling this story about my friend, but I was distracted by the thought of avoiding technical debt, or actually how you shouldn't. Then I completely lost focus and talked about how technical debt isn't really the bad guy; fancy cars are.
Back to the story though. My friend was telling me about his situation, and his application, but more than that, I began to feel something coming through the phone that I didn't expect. It's the change in voice someone begins to make when they are up to their shoulders in tar and they aren't sure how much longer they can hold out. I hoped I could help him.
My friend told me about this monolithic application, a company staple, the most important program in their world. It did everything for everyone. No customer was ever declined. No marketing directive was ever shot down. No timeline was ever exceeded.
Guess what comes out of that kind of meat grinder?
Dozens of developers piling feature upon feature over an old site with tight deadlines and owners who don't understand programming, delivers spaghetti almost every time. Maybe even mashed potatoes, if there is such a thing in programming terms. This is an incubator for technical debt.
So, true to my style, I unloaded everything I could think of about technical debt, like a verbal shotgun. It was 8 months of technical debt knowledge dropped into 3 minutes of monologue. Here are the big takeaways I tried to leave him with:
First, understand technical debt. You are drowning in technical debt interest payments.
Technical debt may be the problem, but more important, technical debt is the conversation. The conversation flows up to your boss and their boss. This conversation is what can bring change back down to you.
You have to refactor. The longer you go, the more you have to do it and the bigger it has to be.
If you can't get time to refactor, you have to take that time. It is part of your job, it has to be. I'll talk more about that later.
Last on this list, remember the golden rule of technical debt: make it visible. Write it where you can see it.
It takes a champion to do what my friend needs to do. I hope he can do it.
posted under category: Software Quality on March 30, 2011 by Nathan
So I was trying to tell the story of my friend who called me up a few weeks back. I got distracted by the question of should you even try to avoid technical debt. Like I said, he has a lot of technical debt.
Remember that technical debt is a way of describing certain problems in programming.
See, there I get stuck on another question: Is technical debt even bad? Is it really just problems? I mean, it is, right? It represents a hole in our program that needs to be filled. The truth is, technical debt isn't the problem, your application is the problem. Technical debt is just a metaphor.
Of course no metaphor would be so damaging if it were only a figure of speech. The phrase "technical debt" is actually very valuable. The real strength lies in its ability to cross boundaries, into management and ownership and finance.
Back in the land of nerds where I dwell, where technical debt has a face, it describes a gap, not necessarily a problem. It's the distance between a perfect application and an app that basically mostly sort of kind of works. Debt is the effort that will get me where I want to be.
My first car was a 1989 Geo Metro. Stick with me here.
I wanted a Lamborghini. I got a Metro. The Metro got me across town, and those 3 cylinders had to work overtime, but there was a noticeable gap between my Metro and the V12 I was hoping for.
Will I ever get that Lamborghini? I don't want to make the effort to come up with those payments, and I don't really need to go 0-60 in 3 seconds - I can't imagine what good that would do.
Will I ever get my application perfected? I don't want to put in the untold hours and make the sacrifice to remove every ounce of debt from my system, and it's ok if there's room for improvement - I can't imagine not having a little room.
Maybe the real goal should be something in the middle, where it's affordable and has enough cup holders. Every application is different, and every working environment is different, but almost none of us are going to get that Lamborghini. My friends, make your applications a Camry or an Explorer. Something that's nice but not impossible.
I'll try to pick up the conversation with my friend next time.
posted under category: Software Quality on March 17, 2011 by Nathan
The title of this post pretty much speaks for itself. Dave interviewed me for my upcoming CF.Objective() presentation in May. If you were wondering what "Holistic Program Quality and Technical Debt" is all about, this is an easy way to find out.
You can listen to the interview online - my interview starts around 59:40 and goes through the end of the show (about 24 minutes).
Of course, if you are into CF and programming, you should subscribe to the whole thing and listen regularly.
posted under category: Software Quality on February 16, 2011 by Nathan
I had a phone call a couple weeks ago with a good friend who was chest-high in technical debt. I was thinking of writing about how to avoid technical debt, but instead I found myself questioning if you should avoid technical debt.
Oh, that may not be a familiar term. I wrote about it last year, but the basics are like this: technical debt is a way of explaining how some problems in software development have a serious monetary drawback. Like any debt, it's usually something you take on by a couple of decisions, and if you're not careful, you end up bankrupt. Most money terms apply here, all puns are usually intended.
So the question comes to me, should we even avoid technical debt? Is it strange that the answer is a definite "No?"
Backing up, we need to really define technical debt. That takes us to the discussion between Robert "Uncle Bob" Martin and Martin Fowler. Uncle Bob says A mess is not a technical debt and Martin Fowler rebuttes with the technical debt quadrant. I think that Mr. Fowler has it right, especially when he reminds us that it's a metaphor, and it applies to both intentional and unintentional debtors.
When I realized that technical debt is just a metaphor, it becomes obvious that it isn't a villain. You can choose to take on technical debt. In fact, many of us do it, but maybe we don't realize it. We do it when we cut corners. We do it when we make our navigation static even though we know it should have been dynamic. We do it when the budget gets in the way of our architecture. We do it when we let our customers or managers push us around about adding features and we don't push back on refactoring. We do it when we know we'll lose a job if we bid too high. Sometimes it's just necessary. Either we take on technical debt, or we feed our kids ramen for a month.
Avoiding all technical debt is not an option. Reducing the amount you take on can be done by refactoring as you go, but sooner or later you will have to do it. When that time comes, don't shy away! Cut that corner! Make the customer happy! Feed your family! Just make sure you follow the golden rule of technical debt:
Keep Your Technical Debt Visible
It should written on the whiteboard. It should be high priority in your bug tracker. It should come up in meetings often. It should be in your email signature. (Ok maybe not that last one.) If you keep it in sight, your management will hear and know about it. Your customers will know about it. It will stay near the top of the to-do list until it's fixed. As soon as you lose sight of it, everyone else will be eager to forget it. You will never be able to squeeze it in without a lot of pain. When you keep it visible, you will keep yourself sane.
posted under category: Software Quality on September 2, 2010 by Nathan
A couple weeks ago, I had the honor of doing a presentation for the Philly CFUG. The subject was about as dry as I could think to make it, but I think it went off pretty well. Being self-critical after listening to it, I have a pretty flat, monotonous voice, but otherwise, the content is good, IMNSHO.
Here's a link to the preso (thanks Adam): Holistic Program Quality and Technical Debt
posted under category: Software Quality on May 25, 2010 by Nathan
Technical Debt has been talked about a lot, but I have been thinking about pretty hard lately. Here's the concept: Make economic analogies for application development to help you (and especially your boss) understand code quality.
Let me lay it down so you can pick it up.
When you write code, you're making a monetary investment, be it in your own time or knowledge, there's been money spent. If it's not literal money, suspend your disbelief for a second when we pretend that your work is worth some arbitrary amount of cash. There's an amount you should spend to complete a given application, especially to do it right.
Now, when you take a shortcut in your programming, you save money up front. You can finish faster, but you're borrowing time, thus borrowing money. Eventually, that money has to be paid back, and you will pay it back.
Paying back technical debt means refactoring the application to undo those coding shortcuts, adding unit tests, making the application maintainable and adhering to best practices. The closer you are to the time you acquired the debt, the easier it is to pay it off. Think of credit cards; pay those off at the end of the month or else you accumulate interest.
Oh, and it's the interest payments that will really kill you.
Until you pay off that technical debt, every time you touch the application, you have to pay interest. In the past, you borrowed time from the future. Now the cruft in the code makes any subsequent task take longer. That is your interest payment. It comes off the top and you never pay it off until you remove the crufty code.
Code with high technical debt tends to be brittle. One change means everything breaks, so we spend our time being extra careful, or by accidentally creating a wide range of errors across the application. Interest makes programming harder, and harder is not what we need.
posted under category: Software Quality on February 20, 2009 by Nathan
I have this open-source toy project I've been playing on, SQL Surfer. It's a little web-based SQL IDE with an amazing build process. How could I not share it with you?
The idea was pretty simple. When I work on my program, I want to see it like a normal program, multiple files, separate CSS and JS, included cfms, conditionals based on the server version, but when I release it, I want it in one, single file: easy to distribute, easy to throw on a server, no install, no folder structure to preserve, no dependencies to remember. It seems that basically no one has ever done this, but it's not too difficult.
The way it works, is, when I get to a point of wanting to make a release, I just run the Ant build file. One click. Ant does what it does best, preps the build, cleans out the output folder, in the end, it zips the results and whatnot, but I couldn't pull off the dynamic file embedding with plain Ant XML, so that's where Groovy comes in. I've got Groovy searching for included files (cfinclude, script, style, etc) and embedding the files in place of including them. Plus, while I was mashing files together, it seemed logical to compress them, even if just a little.
The result is a mashup of my entire application, nearly obfuscated into a single, gigantic, single-file, working application.
Actually, it goes a little further still. I wanted to have a version to take advantage of ColdFusion 8's new <cfdbinfo> tag, which, by the way, is awesome, but I also wanted to stay compatible with CF 6 and 7, even with a lesser experience. When you type a new CF tag into an older version, you get a compile error. That makes it hard to develop and hard to release. My solution was to drop in
<cfif left(server.ColdFusion.productVersion,1) GTE 8>, followed by a cfinclude, css or js reference, then, Ant makes 2 files for 2 releases and my Groovy build file has 2 functions, removeCF8 for removing the if blocks, and removeIfCF8 for just the if statements and embedding the files.
The actual result is 2 files, all mashed up and working perfectly for different server versions.
Going overboard, I also embed EditArea, for the code editor. Of course EditArea doesn't care about an all-in-one-file project. I use Groovy to insert EditArea into the main file as well as change EditArea itself, at build time, to point to the embedded parts in the CF application instead of its own directory structure.
Taking it beyond reality, I realize now that I could probably include a framework like Barney Boisvert's excellent FB3 Lite, and then potentially images in b64. The only challenge that this approach leaves me with is CFCs. I love my objects, but I haven't figured out how exactly to embed them.
Ouch. Even just typing that last sentence makes my head explode with ideas. Taking it way too far, I could probably embed the components I use onto the main file, stripping out the cfcomponent tags, then to recreate them, create instances of the generic coldfusion component, WEB-INF.cftags.component, and re-constitute them with duck typing and method injection... Yeah, maybe some day, if I get really into it.
You can check out the source code right now and compile it yourself, but to hear about it first hand, visit the AZCFUG Feb meeting, 2/25/09, where I'll be showing it off. If you miss it, don't sweat it, I'll have the slides out by the next day. I'm undecided on having a connect meeting for it. If we choose to, I'll post it up here.
posted under category: Software Quality on September 6, 2008 by Nathan
Selenium is an open source, simple web site testing and automation platform. I say it's a "platform" because it can't really be classified as just a tool, a language, API or a full application, but rather to a degree, all of these.
(continue reading) (discuss!)
The basic idea is to create, using very plain, easy HTML, a repeatable script that is then executed by a Selenium runner.
I recommend using the Selenium IDE to create your first script. It's an add-on to IE and Firefox that records your clicks. Save it to a file (it generates plain HTML) and you can repeat it later on. It won't always work perfectly, however, so you can edit it with any text or HTML editor. You just need a Selenium language reference to get by.
Selenium actions and accessors are the language you program your scripts in. Generally, you would use the
type to enter values into forms and things like
assertText to make sure certain content exists.
One part of using the language you will have trouble with is selecting elements. Hopefully, you use uniquely named text links everywhere. If not, Selenium supports XPath very nicely. For me, it's a matter of sprucing up my XPath skills, something I wanted to do anyway.
Selenium scripts consist of a table with three columns. The first column is the action to perform. The second column is like the first argument of a function, usually this is the item to perform the action on and is almost always used. The third column is like the second argument in a function, usually the content of the action like the text to enter into the form item from the second arg. This column is not always used.
Selenium scripts must run within your web browser, however, different tools to automate your browser exist:
The Selenium IDE will run a file off your disk. This is perfect for single scripts and quick browser automation.