https://martinfowler.com/articles/testing-culture.html
It’s likely that the programmer who wrote this algorithm the first time did execute the program to check for errors in the new code. Most programmers will run a program with some sample inputs to verify that it’s doing what they think it should do. The problem is that these runs are often ephemeral and thrown away once the code is working; an automated test captures those runs as a permanent double-check.
- Yeah many time (often in legacy) code that I write some precious test and then have to clean it after done to push to Git.
That permanent double-check is important here: We don’t know exactly how that rogue second goto fail got into the code; a likely reason is that it was the result of a large merge operation. When merging a branch into the mainline, large differences can result. Even if a merge compiles, it can still introduce errors. Inspecting such merge differences can be time-consuming, tedious, and error-prone, even for experienced developers. In this case the automated double-check provided by unit tests provides a fast and painstaking (yet painless!) code review, in the sense that the tests will likely catch potential merge errors before a human inspects the merged code. It’s unlikely the original author introduced the "goto fail" bug into the code, but a suite of tests doesn’t just help you find your own mistakes: It helps reveal mistakes made by programmers far into the future.
Many bash technique here:
$ curl -O http://opensource.apple.com/tarballs/Security/Security-55471.tar.gz
$ curl -O http://opensource.apple.com/tarballs/Security/Security-55471.14.tar.gz
$ for f in Security-55471{,.14}.tar.gz; do gzip -dc $f | tar xf - ; done
# Since diff on OS X doesn't have a --no-dereference option:
$ find Security-55471* -type l | xargs rm
$ diff -uNr Security-55471{,.14}/libsecurity_ssl
diff -uNr Security-55470/libsecurity_ssl/lib/sslKeyExchange.c
Security-55471.14/libsecurity_ssl/lib/sslKeyExchange.c
--- Security-55471/libsecurity_ssl/lib/sslKeyExchange.c 2013-08-09
20:41:07.000000000 -0400
+++ Security-55471.14/libsecurity_ssl/lib/sslKeyExchange.c 2014-02-06
22:55:54.000000000 -0500
@@ -628,7 +628,6 @@
goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
- goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
diff -uNr Security-55471/libsecurity_ssl/regressions/ssl-43-ciphers.c
Security-55471.14/libsecurity_ssl/regressions/ssl-43-ciphers.c
--- Security-55471/libsecurity_ssl/regressions/ssl-43-ciphers.c 2013-10-11
17:56:44.000000000 -0400
+++ Security-55471.14/libsecurity_ssl/regressions/ssl-43-ciphers.c
2014-03-12 19:30:14.000000000 -0400
@@ -85,7 +85,7 @@
{ OPENSSL_SERVER, 4000, 0, false}, //openssl s_server w/o client side
auth
{ GNUTLS_SERVER, 5000, 1, false}, // gnutls-serv w/o client side auth
{ "www.mikestoolbox.org", 442, 2, false}, // mike's w/o client side auth
-// { "tls.secg.org", 40022, 3, false}, // secg ecc server w/o client side
auth - This server generate DH params we dont support.
+// { "tls.secg.org", 40022, 3, false}, // secg ecc server w/o client side
auth
{ OPENSSL_SERVER, 4010, 0, true}, //openssl s_server w/ client side auth
{ GNUTLS_SERVER, 5010, 1, true}, // gnutls-serv w/ client side auth$ curl -O http://opensource.apple.com/tarballs/Security/Security-55471.tar.gz
$ curl -O http://opensource.apple.com/tarballs/Security/Security-55471.14.tar.gz
$ for f in Security-55471{,.14}.tar.gz; do gzip -dc $f | tar xf - ; done
# Since diff on OS X doesn't have a --no-dereference option:
$ find Security-55471* -type l | xargs rm
$ diff -uNr Security-55471{,.14}/libsecurity_ssl
diff -uNr Security-55470/libsecurity_ssl/lib/sslKeyExchange.c
Security-55471.14/libsecurity_ssl/lib/sslKeyExchange.c
--- Security-55471/libsecurity_ssl/lib/sslKeyExchange.c 2013-08-09
20:41:07.000000000 -0400
+++ Security-55471.14/libsecurity_ssl/lib/sslKeyExchange.c 2014-02-06
22:55:54.000000000 -0500
@@ -628,7 +628,6 @@
goto fail;
if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
goto fail;
- goto fail;
if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
goto fail;
diff -uNr Security-55471/libsecurity_ssl/regressions/ssl-43-ciphers.c
Security-55471.14/libsecurity_ssl/regressions/ssl-43-ciphers.c
--- Security-55471/libsecurity_ssl/regressions/ssl-43-ciphers.c 2013-10-11
17:56:44.000000000 -0400
+++ Security-55471.14/libsecurity_ssl/regressions/ssl-43-ciphers.c
2014-03-12 19:30:14.000000000 -0400
@@ -85,7 +85,7 @@
{ OPENSSL_SERVER, 4000, 0, false}, //openssl s_server w/o client side
auth
{ GNUTLS_SERVER, 5000, 1, false}, // gnutls-serv w/o client side auth
{ "www.mikestoolbox.org", 442, 2, false}, // mike's w/o client side auth
-// { "tls.secg.org", 40022, 3, false}, // secg ecc server w/o client side
auth - This server generate DH params we dont support.
+// { "tls.secg.org", 40022, 3, false}, // secg ecc server w/o client side
auth
{ OPENSSL_SERVER, 4010, 0, true}, //openssl s_server w/ client side auth
{ GNUTLS_SERVER, 5010, 1, true}, // gnutls-serv w/ client side auth
...
https://queue.acm.org/detail.cfm?id=2620662
How Could Unit Testing Have Helped?
As opposed to the case of the “goto fail” bug, there is no need to extract a new function: both dtls1_process_heartbeat() and tls1_process_heartbeat() are already good-sized units that don’t require a large amount of complicated setup to get under test. We can get right to the same questions posed earlier in the context of “goto fail”:
What is the contract fulfilled by the code under test?
What preconditions are required, and how are they enforced?
What postconditions are guaranteed?
What example inputs trigger different behaviors?
What set of tests will trigger each behavior and validate each guarantee?
Given the power of modern version control systems and the increasingly-common practices of forking, merging, and cherry-picking, tests have become more important than ever to guard against unintentional changes, especially changes leading to a regression of a known catastrophic bug. The apparent removal of a regression test during a cherry pick or a merge should set off alarm bells, even more so if the test was included in the same change as the fix, as the fix could become undone as well.
...
The buck stops with the code review process, whereby a change is accepted for inclusion into the code base by the developers who control access to the canonical source repository. If unit tests are not required by a code reviewer, then cruft will pile on top of cruft, multiplying the chances of another "goto fail" or Heartbleed slipping through. As was perhaps the case with "goto fail", the development teams at many companies are focused on high-level business goals, lack any direct incentive to improve code quality, and perceive an investment in code quality to be at odds with shipping on-time. As was the case with Heartbleed, many Open Source projects are volunteer-driven, and the central developers are short on either the time or the skills required to enforce the policy that each code change be accompanied by thorough, well-crafted unit tests. No one is paying, rewarding, or pressuring them to maintain a high level of code quality.
The Costs and Benefits of a Unit Testing Culture
While unit testing can greatly reduce the number of low-level defects, including defects as high-visibility and high-impact as "goto fail" and Heartbleed, and have a positive influence on other aspects of code quality and the development process, building and maintaining a unit testing culture comes at a cost. There’s no such thing as a free lunch.
On the other hand, be aware of the saying: "There is nothing more permanent than throw-away code." The trade-off is that the more features are implemented without accompanying tests, the more Technical Debt a team builds up that must be repaid later. Unit testing can be difficult if you don't design for testability from the start—using dependency injection, writing well-defined classes that focus on one thing, and so forth. It is up to the team to gauge the acceptable limits of such debt, and at which point it must be paid to avoid an even more expensive rewrite once maintenance and new feature development grow too cumbersome.
Despite this difficulty, integrating new features was integral to the success of Google as a business. The barrier that was stopping people from making changes as rapidly as possible was the same that slows change on most mature codebases: a quite reasonable fear that changes will introduce bugs.
Fear is the mind-killer. It stops new team members from changing things because they don't understand the system, and it stops experienced people changing things because they understand it all too well.
Furthermore, the mitigation of fear led to the expansion of their joy in programming, as they could see tangible progress being made towards exciting new milestones without being held back by chronic outbreaks of high-priority bugs. The impact on productivity of high morale, based on the ability to remain in a state of creative flow, cannot be overstated. While I was at Google, the GWS Team exhibited the ideal testing culture, integrating an enormous number of complex changes from outside contributors while making their own constant improvements.
Tight Feedback Loops
Executable Documentation
Accelerated Understanding
Faster Bug Hunting
Are You Experienced?
Get Your Hands Dirty
No Test Is an Island
Other Useful Tools and Practices
Static Analysis/Compiler Warnings
Modern Languages
Open-Sourcing
Comments
Post a Comment