Why aren't you testing?
Courtenay : December 15th, 2008
Those of you passionate about test frameworks or writing your own testing libraries can skip this article.
The rest of you, who don’t test: you’ve heard about it, you’ve seen us ranting about it, but, if you have a web application written in Rails and you don’t write tests for it, how do you know it works? (I’m guessing: refreshing it in your browser). How do you know it works for other people, too? If the company for which you work doesn’t have a testing culture, why don’t you step up and insist? Take a weekend and learn about it. Do the peepcodes, then evangelize at your company.
I’m stunned at how many Rails development companies out there just don’t test. At all. And worse, how many people apply for Rails jobs without any tests for the code they included as part of their application. If an employer gets two resumes, one where the coder has submitted a few files of code with tests, and the other which is just rails scaffold in test/, who do you think they’ll hire?
23 Responses to “Why aren't you testing?”
Sorry, comments are closed for this article.
December 15th, 2008 at 04:10 PM
Before I get into this I want to mention I am not a full-time Rails developer - not even part time. I play around with Rails when I have an idea. I just love the way it works and I love the whole process of writing a Rails app. I am a front-end developer (xhtml,css,javascript,actionscript) and have been for 10 years. My testing has always been refreshing the browser because there isn't an other way (outside of actionscript).
As for testing in Rails - I honestly don't know what and when to test. Do I need to write a test to make sure I have a subject in my email even though the validation is taking care of it for me? Do I need to write a test to make sure any of the built-in validators are working? Do I need to write a test to make sure my getunpublishedarticles is actually giving me unpublished articles, even though if it's giving me anything at all that means my method is working?
I'm sure the answer to all of these questions is yes. If I watched a video of someone writing a Rails app start to finish, including writing the testing code, then maybe I'd see where it fits in other people's processes. Maybe then it would make more sense to me, after all, watching the videos from Peepcode and Railcasts are what got me into Rails in the first place. Neither of them have gotten that into testing while writing another app. They focus on testing and testing alone which doesn't help someone completely unfamiliar with it.
December 15th, 2008 at 04:16 PM
I Feel that testing breaks 'flow' (as in http://en.wikipedia.org/wiki/Flow_(psychology) ). Might be because I'm not used to doing it and still have to think too much about it.
I see the value of testing when working on a project with multiple developers that might be refactored and maintaned by other developers down the line. But for a personal let's-hack-this-together-fast kind of project, I feel testing causes too much overhead.
On the other hand, when working in a noisy environment with lots of interruptions (and thus low or no flow), I noticed that taking the tdd approach actually helps to focus and be more productive.
December 15th, 2008 at 04:38 PM
Most won't admit it, but I have done this in the past. Slowly I have changed my ways.
I had to overcome a psychological barrier. I work by myself in a one man shop. My projects are internal and are never viewed by others. Pair programming and peer review are non-existent for me. Honestly, I had built up some bad habits in 15 years of programming.
What changed?
It was not pride, applying for gigs, peer pressure or rants by developers I respect - it was the calculation of lost productivity and opportunity costs.
When you do NOT test up front, the loss of time and momentum when you stop to find and fix bugs is exponentially greater than writing the tests up front.
It was that simple for me.
Peepcode was a blessing when I dove in the testing waters. You can carry them around on your iPhone and do refreshers to retrain yourself.
December 15th, 2008 at 05:41 PM
What really sucks is when you have to take over a project when no tests were ever written (or worse yet, when the only tests that were written were the scaffolding tests, and none of them were ever updated to the ongoing development of the app)
So be nice, support future development on your app - write tests :)
Piet: No way man, testing reinforces the flow. Once you don't need to think about it as much (i.e., with flow it would be a too-challenging task), then it actually helps move you along from idea to idea.
December 15th, 2008 at 07:04 PM
I would love to test. Really, I would. I see the value in it, I'm totally onboard with the TATFT mindset, and I know it would save me time and energy. But I haven't found any tutorials or resources that adequately break it down for a total beginner. I see the religious wars around Test::Unit vs. Rspec vs. Test::Unit + Shoulda vs. Cucumber, and using factories vs. using fixtures, and mocking, and it's all pretty intimidating. Working through the tutorials quickly becomes a morass of "this is a functional test, and you need to do it in these cases, on this type of code; this is a unit test" ... and so on. That's all well and good, but despite all of the fractured divisions, from the outside, it's this huge, oppressive, monolithic beast. For as much as the community talks about agile development, you'd think there'd be some agile tutorials that would help you get started.
Like Jay says in the first comment, I'd love for someone to show a screencast that explains the goal, the process, and the result, in clear, step-by-step terms.
In general, I think this has been a real problem for Rails: tutorials that assume more knowledge than a lot of beginners have, or that use outdated modes and methods (often that don't even work with the latest version of Rails). I appreciate the new guides that Pratik and everyone have been doing, and I'm hopeful that the testing guide will be helpful. But my early glances at it have left me confused.
But to get back to the question at hand: I would love to write fully-covered code; I would love to write according to TDD/BDD best practices. At the moment, though, it feels like there's a hidden door that everyone who tests has found and walked through and gotten it, and there are a bunch of us on the outside that would love to join the party, but don't know where to find the door. I know: the door is really more like a tunnel, and you only pass through the tunnel by practicing. Of course. The problem is that from the outside, there are a lot of different holes that look like the entrance to "the tunnel", and I've made enough false starts that I'm weary of exploring.
I want to be there with you. I'm going to keep working on it. At the same time, I'd appreciate any pointers you have for people who are willing to put in the effort, but have absolutely no idea where to begin.
December 15th, 2008 at 08:34 PM
I've been trying to 'evangelize' my peers who do rails development and I feel like the holdup was the same for all of them. If you're not already doing it, it never seems like you have the time to learn it before you're back operating smoothly again. The other posters are right, there's no good guide to get you into doing it.
There don't seem to be any good resources that really walk you through it in way you can connect to your own app. In the end, the only people from my peer group I've gotten to use it (at all) are the ones that I sat with, peer programming, walking them through introducing a new feature into their app BDD style. We need some sort of tutorial that can walk people through it in way that makes it 'click' in a broad way.
I'm lucky that I started rails WITH rspec, so I really didn't have a choice but to just incorporate it. Even now it's hard sometimes to maintain the discipline to test first. I certainly understand how for those who don't but want to, it's really hard right now to find the tools to get you into the game. Especially when so many articles argue about what the best practices are, whether best practices really exist, and whether it even matters. I've tried myself to write better guides, but so far haven't been able to make one as effective as it should be. Hopefully that's just a matter of time for our community!
December 15th, 2008 at 09:07 PM
I'm going to go ahead and say it. Testing is a tool. It is not a religion. Test when and what makes sense. You don't have to test everything but you may need to. It depends on a number of factors such as:
Desired level of reliability
Most projects are internal or personal and some failure is preferred over the higher costs necessary to ensure the failure doesn't happen. Someone who is good at tests can have tests without increasing costs too much but let us not lie to ourselves. In the long run test probably decrease cost but most projects never make it to the long run. Many projects are developed and then left alone with just perhaps a minor tweak or two down the road. In these cases test do increase cost and therefore may not be desirable if a certain level of failure is acceptable and the backup/restore process is good.
Current Phase in a Project
Some projects are what I call exploratory. This is when the client/boss really doesn't know what the hell they want. They think they have an idea of what they want but once you implement version one you quickly start getting change orders to the basic plumbing of the site. In these cases I tend to keep the controller tests light (as the flow of the app is likely to have major changes soon) and even my model code I may not test as fully as a version 2 or version 3 phase of a project.
If you overdue testing on a exploratory idea that only means you will have to throw even more code away when you make the major changes you will almost certainly need to make.
Development Process
Are you likely to hand the code off to some set of unknown people to maintain. Or are you likely to continue to own the project (or at least have a small number of other people to own the project). When I say "own" I mean take responsibility to maintain it.
If there are going to be many hands touching your app after you hand it off (like an open source project) then you need good tests because those people will just be concerned about the area of the app they are making their adjustment in. They want to be able to make their adjustment, run the tests and know they didn't fuck anything up.
On the other hand if you (or someone else) is going to take responsibility of the app then you will may not need as much testing because whoever owns the app will more intuitively know what will fuck the app up and what will not.
You might be saying "What if down the road our little app gets bought by Big Corp. and we have to hand it off?" I would say consider what is likely to really happen and take the appropriate action. Don't code to much for "what ifs". It is possible (and fairly easy) to add fuller tests later if you decide that is now important.
Type of code you are writing
This is the biggest place I think people overdue testing. I have seen stupid stuff like testing the following code:
The test code will then verify that calling @post.comments actually returns the right comments for the given post. WTF?!?! If "hasmany :comments" is not functioning as expected then we have a problem in the framework not web app. The test to verify "hasmany :comments" works properly should be in Rails (and is) not your app. If it is broken then you need to fix it in Rails (and implement the tests there).
If you are just using a 3rd party library then your app should assume that library is tested and not test something the library should be testing for (if it is not and you want to better your reliability then put your test in the 3rd party library). My general rule is if the code is declarative then you are less likely to need to test. If the code is procedural then you are more likely to need to test. So if you have a ActionMailer method that looks like:
Do you need to test the above code? Probably not. You may want to test the output of the template if it uses "@user" is some complex way but if we are just considering the above code it is all declarative and well tested already by Rails. So don't bother doing things like:
You are just wasting time and adding to the complexity of your test code and rigidness of your application. You are basically testing to make sure you typed the characters in correctly. What does that prove? If you mistyped them in the actual code then you are likely to mistype them in the test code because you copied and pasted the subject string. So now you are testing to ensure you can copy and paste correctly. Now these are not hard and fast rules. For example consider:
Do you test that? Maybe. Although it is declarative it is also complex. The more complex the declarative statement is the more likely you will need to test. Again it depends on the app and your goals.
Final Words
My final thoughts on testing is this. Don't assume because you have full testing that your app works. Most likely you are only testing the thing you can think might go wrong but most likely the things you don't think of are the things that will actually go wrong. Testing is good for speeding up integration and ensuring you do not have regressions. But is is not proof of a working app (the absence of a failure does not prove correctness).
Also don't take my post to mean that you shouldn't test. I test and test quite a bit. I average about 1:0.8 code to test ratio so that is still a decent amount of testing. I just think you need to think about what you are testing and test smart. Don't test because you feel like you will be a bad boy if you don't. Test because it helps you develop your app or achieve some reliability goal your app has.
Obviously others may disagree and I have no problem with that. But don't blindly implement thousands of tests just because someone makes you feel bad for not testing. Test because you think it is providing a benefit to your project.
December 16th, 2008 at 02:44 AM
Why am I not testing?
Testing is a bit of a chicken and egg problem. People who test have the experience to write "good" tests, that benefit what you're doing. People who don't have that experience yet will write tests that won't really help them. Since they're not testing "right", they don't get any (or much) benefits from testing, and will soon give up on it.
The problem here is that in order to write good tests, you first have to write lots of bad tests that won't really help you. Having a teacher guide you is a big, big help here, but if you're on your own you are going to waste a lot of time for essentially nothing (Like learning a new language, you're going to reap what you sow, but while learning people aren't really going to think of that). And that aspect tends to get either ignored completely or swept under the rug by testing advocates (in my experience).
And unfortunately this is an area where books don't help that much, since testing isn't something you can really describe in a book, but must experience yourself.
So why am I not testing? Because testing right is hard. Because it's something that might save your bacon one day, but doesn't give instant gratification like learning a new programming language. Because it just takes so damn long for me to write tests, time that I feel could have been better spent somewhere else afterwards.
I am learning to test. One step at a time. But I don't think I'll be able to write "good" tests any time soon, tests that would one day actually save my bacon.
December 16th, 2008 at 07:51 AM
At the risk of getting too plug-y, check out railsprescriptions.com -- this will be the guide that you are hoping for, coming in January. Leave a comment there if there are specific things about testing that you want covered. Watch the blog, too, it'll be covering some ideas on integrating testing.
Also, my book Professional Ruby on Rails has nearly all examples in the book written test-first, and I think it's a pretty good guide to managing test-first development when working in a test-driven style.
Thanks,
Noel Rappin
December 16th, 2008 at 09:00 AM
You said Tender would be ready two weeks ago! I'm really looking forward to using it.
December 16th, 2008 at 10:34 AM
Why is this even an issue? Why aren't you always testing? If you have enough time to reason why you aren't testing, you have enough time to at least attempt to test something.
Some people say it is hard, and they don't want to do it, because they don't want to write bad tests. This is crazy talk. This is the same as saying you don't won't to go to the gym and lift weights because you can't benchpress 300 pounds.
A better way to to approach testing is make your spec/tests your output. Play it like a game. Right failing specs. Then write code to make those specs pass. Then refactor the code/specs you have just written to remove repeated statements or even make methods easier to read.
When you try to apply testing to code you have already written, you'll naturally will fill like you are wasting times. I bet that your tests will look exactly like your code. Then I bet you'll come to the conclusion that writing tests is a waste of time. If this sounds like you, take a deep breath, and realize you are doing it wrong. Try test first development. With practice, you'll see how you can become even more efficient.
And if you aren't sure about when you should be testing.
December 16th, 2008 at 10:58 AM
Okay, to all the folks (those who've commented and those who haven't) worrying about "best practices" or excusing themselves because they are beginners...
I sympathize, I really do. The "state of the art" for testing touches not just on testing's best practices, but on the best practices in software architecture as well.
If you're a beginner, trying to learn from the guiding lights in the testing world can be like shining bright light on just how much you suck.
But you need to remember two critical facts:
1 - Every single Testing Guru used to suck just as bad as you. 2 - You're not letting your "beginner" status excuse you from writing production code.
Yeah, testing is hard. Sometimes I think it's so hard that it can't actually be taught - it must be learned.
Maybe it's going to take you two years to become really good at testing. Sounds like a long time doesn't it?
Well, your choice is simple, you can start the two-year clock ticking today and do the best you can with the skills you have - your tests may suck but you're always looking for ways to improve your code anyhow right?
Or you could continue as you are, searching for a tutorial that speaks to you and making excuses all the while.
December 2010 is going to arrive either way.
December 16th, 2008 at 11:23 AM
Quick question, to Courtenay, Bryan, Eric, Brad, Trevor, and others: How did you initially learn testing? Was it through pair programming? Was there a tutorial? A book? I'm not talking about getting better at testing. I know that comes from practice. But there was some point where you went from not having any idea where to start, to knowing how to write tests for your code. What flipped the switch?
If someone showed you how to do it ... what would you suggest to someone isolated from other programmers, who isn't in a pair-programming environment? What's the best way to approximate pair programming? Is it even possible?
If you read a tutorial / book / screencast ... which one was it that made it click? Which ones did you try that you'd recommend against?
To Noel: I'm looking forward to Rails Prescriptions. Thanks for mentioning that.
December 16th, 2008 at 11:36 AM
Hmm. I think my comment just got eaten.
Could those of you who TATFT comment with how you initially learned how to test? Not how you got better (I know that comes with practice), but how you went from "not getting how to test" to "getting it." Thanks.
December 16th, 2008 at 02:53 PM
There is a habit in the greater web developer role, to not write tests. I think this can mostly be attributed to the lack of decent frameworks.
Once you have been coding for years without tests, and it's always new code, it seems like a lot of time spent verifying what you already think you know - that your code works. And when it doesn't work, you do a thing called debugging which you already know. Even if it takes a whole day to find your php bug, it seems like if you spent that time writing tests then you might not have picked up the bug anyway, and would be down a day or two. And testing a codebase exhaustively is near impossible.
Most of the people who feel this way are either LAZY or HAVE NEVER MAINTAINED SOMEONE ELSES CODE.
What testing gives is another document for your code, that explains how it should function, in an executable format. So instead of checking off test cases on a test page manually, every time you make a change you can just run the tests in about a second! This cuts down your testing time considerably, over the life of the project.
With ruby, you have awesome testing frameworks. Rspec is like a gift from the gods. But it is poorly documented. Try searching http://rspec.info on how to say a function should throw an error.
Until you have a list of examples from trial and error, the learning curve is steep.
December 17th, 2008 at 10:31 AM
Charlie,
I think I wrote my first unit test more than 10 years ago and to be perfectly honest, I regularly question how well I'm "getting it" :-)
But I think I can identify the point between "not testing" and "TATFT" that you call "getting it", and I think I know how you get there too.
It starts with writing tests - lots of them. Write the best tests you can, with the knowledge you have.
Don't concern yourself too much with stuff like "best practices" - they will rob you of valuable learning opportunities.
After a while - and no, I don't know how long - you will come to rely on your tests. You may not actually be doing test-first development but you'll at least have an implicit expectation that the production build of your software should always pass all tests.
Here's where it gets interesting: after a while you'll come to despise your tests. At best you'll see them as a necessary evil. At worst, you'll see them as a cumbersome mess that's starting to get in the way of getting 'real' development tasks finished.
You know you can't go back to having no tests at all, so you start looking for ways to improve the situation. "Maybe we change our code structure to make testing simpler ... or perhaps we ought to look at a different testing framework ... wait a minute, half this stuff doesn't actually add any value!"
You refactor your tests and they suck much less than they used to.
At that point you may not proclaim to know "The One Right Way To Test" but you sure as hell know some of the wrong ways and you know how to improve the situation.
That's what "getting it" looks like.
December 18th, 2008 at 07:01 PM
Trevor - Thanks for writing that up. I would (of course) love to hear others' input on that as well, but I do really appreciate your thoughts. Both of your posts on this thread have been helpful.
December 24th, 2008 at 03:53 PM
Eh.
I know! I feel guilty about it all the time! I have many acquaintances who are all into it and etc and every other blog post is TATFT and I even know the difference between testing frameworks from all the blogs, but holy shit it's hard enough to find the focus to bang out the extra feature or two let alone sit down and learn a whole 'nother framework then break my flow all over the place to do it.
It's like flossing or quitting smoking; i know I should do it but holy shit I cannot be bothered right now.
December 29th, 2008 at 05:07 PM
Wow, whats a great detailed post! Thanks for sharing! HGH
December 30th, 2008 at 08:52 AM
I think Eric Anderson above (with his fancy formatted post! ^_~) really hits it on the head. I don't think you can responsibly get by without a decent amount of unit and functional testing (and integration testing totally rules for helping you to check that certain critical things like account signups and referral processes are working), but Luke Francl gave a really convincing talk over the Summer at Ruby Fringe saying basically that testing itself is unreliable because we're not very good at it, because we write buggy code and can't see all the possible problems.
So, I hope you're not just testing but also refreshing your browser, or you're in for some ugly surprises in your production environment.
January 2nd, 2009 at 08:32 AM
There are a lot of poor arguments against testing. Usually they boil down to one of two things: laziness or hubris.
First, laziness. The kind of laziness I'm talking about here is the opposite of the kind Larry Wall talks about. This is the type of laziness that leads one to avoid a short term cost even though it means that they will incur a much larger long term cost. Writing tests saves time by making code more reliable and more maintainable. Period.
Those of you who say that you don't write tests because it takes too long simply haven't evaluated the full life cycle cost of your decision.
Every time you think to yourself, "Do I need to write a test for this?" the answer is always: Yes!
Secondly, there are people who don't think they can gain anything from testing. This one can usually be attributed to hubris. Again, not the good kind that Larry Wall talks about, but the kind that makes you assume that your code is too good to need tests.
Consider, first, that all code is buggy. The better the programmer, the less the bugs (perhaps), but never zero. You will write bugs. The more subtle (and hard to "test" manually), the harder to find and fix. Good tests will catch more of these bugs sooner.
Second, keep in mind that most code in the real world will have more than one set of eyes on it. Even if you are perfect, other people aren't. Let's face it, you'll always need to protect your code from being screwed up by other people. Tests ensure that no one else will break your perfect codes.
Tests also serve as living documentation (specifications) of the behavior of the system for the poor maintenance person that inherits your code, making their life easier. This includes the 6-months-later you who has to go back and fix a bug.
Now, in fairness, I would like to mention one valid argument against testing. Testing increases the maintainability of code and as such has the side effect of increasing the expendability of the author as well. Since tests are required to ensure code fitness (and to prevent your termination on grounds of incompetence), I suggest a compromise solution: write your tests, but delete them before checking your code in so that no one else gets to use them to take your job.
January 5th, 2009 at 08:05 AM
This is the biggest place I think people overdue testing. I have seen stupid stuff like testing the following code:
The test code will then verify that calling @post.comments actually returns the right comments for the given post. WTF?!?!
I don't think I agree here.
Testing is not only for checking out that the code is working, but also to factor out human errors. Say for example this new 'dick' in the company has commit rights. He comments out for no reason.
# has_many : comments
Ruby is not compiled so there is no way the system knows of this error. Two days later there is this git push and cap deploy. Oops.
This oneliner in shoulda is saving my ass
should_have_many :comments
just my two euro cents.
January 5th, 2009 at 01:18 PM
@Dude - What happens when that same "dick" comments out your shouldhavemany :comments line to keep the code passing the tests. Maybe he thinks the hasmany line really needs to be removed and therefore also the test for that line. So your shouldhave_many line didn't help one bit.
What might help though is other more useful testing where you are actually using the relationship (say in a action on a functional test on a controller). When that functional test runs it tries to use the relationship which is not there anymore and therefore you get an error. So really your hasmany line is already tested without your shouldhavemany test. All you are doing by adding the shouldhave_many line is adding noise to your test, making your tests take longer and increasing the maintenance costs of your app because it has become less DRY.
Just my two cents :)