On the .NET platform, the exception raised reads “System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted <host>:<port>“. 

In Java, the exception is “java.net.BindException: Address already in use: connect“. 

Both exceptions are misleading because they are generally associated with server socket conflicts – not outbound client socket connections.  However, a better understanding of the TCP state machine sheds some light on this behavior – and a solution.

Common Port Conflict Exceptions

Whenever TCP/IP encounters a port conflict, you can expect one of the two following exceptions to be thrown depending upon your environment:

In a C# environment, you’ll see this exception:

System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted <host>:<port>

In Java, you’ll see this:

java.net.BindException: Address already in use: connect

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

at java.net.Socket.connect(Socket.java:519)

at java.net.Socket.connect(Socket.java:469)

If you see these exceptions thrown when a server socket attempts to listen for incoming connections, the cause is obvious: the port you’re attempting to listen on is already in use.  For example, if you bring up a Web server, but another Web server is already running, an exception will be thrown because port 80 (or 8080) is already listening on behalf of another thread or application.

These exceptions are often confusing, however, when thrown setting up a client connection.  Client TCP connections are always assigned an OS-selected port on the local side, so why is the operating system selecting an active port?  The truth is the exception indicates no local port numbers are available to the client.  This misreporting of the error by the OS is half the confusion.

Tuning Local Client Port Range

The problem is two-fold.  First, Linux and Windows make only a certain number of ports available to client sockets – the default is in the range of 1024 to 5000.  Hence, you may have only 3,976 active client connections at a time.  For most systems, this is plenty.  However, in specific circumstances on systems requiring a large number of outbound connections, this limit can be exhausted.

This range, however, can be tuned.  On Windows,  the upper bound for client port assignments can be adjusted using the MaxUserPort DWORD value on this registry key:

HKLM\System\CurrentControlSet\Services\Tcpip\Parameters

Of course, rather than using the cumbersome regedit, you can download IPTuner for free to quickly optimize your Windows IP stack

On Linux, both the lower and upper bounds can be set using the following parameter:

net.ipv4.ip_local_port_range = 32768 65536

How this parameter is set in Linux varies depending upon the flavor and version of Linux you’re using – and you’ll need to restart networking after you change it.

Tuning TCP TIME_WAIT Timeout Value

The second cause of these exceptions has to do with the TCP state model and the way sockets are closed.  Even after a socket has officially been “closed”, it hangs around in a TIME_WAIT state as a safety mechanism to deal with stray packets.  The default wait time on all operating systems, generally, is ridiculously long (240 seconds on Windows).  So, even if an application doesn’t require a lot of concurrent connections, it can still run out of available ports in a high load situation.  If even one connection is repeatedly opened and closed fast enough, you’ll soon have all available local sockets hanging around in a TIME_WAIT state and none available for new clients.

The TIME_WAIT state duration, however, is also tunable.

On Windows, using the same registry key, the TCPTimedWaitDelay value can be used to adjust the TIME_WAIT duration from 30 to 300 seconds.  Of course, rather than using the cumbersome regedit, you can download IPTuner for free to quickly optimize your Windows IP stack.

On Linux, the wait delay is configured using the following parameter:

net.ipv4.tcp_fin_timeout = 30

By decreasing the TCP wait delay, closed sockets spend less time in the TIME_WAIT state and get returned to the pool of available client ports faster. However, to avoid communication problems, do not lower this value below 30 seconds.

Administrator Privileges Required

On both Linux and Windows (even if using IP tuner or regedit), you’ll require administrator privileges to change these parameters.  However, anyone can view these settings to at least verify if they make sense.

• Download IPTuner for Windows here.

However, managing multiple threads and cross-thread communication adds complexity to your code.  Fortunately, the .NET Framework has a useful design pattern applied to its I/O classes which easily enables asynchronous calls.  Let’s take a look at an example.

Async I/O for a DNS lookup 

Suppose you need to lookup the IP address of a host.  The simplest way to do this is to use the System.Net.Dns class:

IPAddress[] hostAddresses = Dns.GetHostAddresses(“www.gavaghan.org”);

A DNS lookup doesn’t take terribly long and, in most cases, the synchronous example above is fine.  DNS servers are highly efficient, and local DNS servers will cache authoritative data to optimize response times.

However, suppose you’re implementing a high performance mail server that detects spam by querying multiple real time DNS blacklists.  For every incoming message, you must execute a dozen DNS operations:

IPAddress[] spamcop = Dns.GetHostAddresses(“22.154.199.213.bl.spamcop.net”);

IPAddress[] spamhaus = Dns.GetHostAddresses(“22.154.199.213.pbl.spamhaus.org”);

IPAddress[] fiveten =

Dns.GetHostAddresses(“22.154.199.213.blackholes.five-ten-sg.com”);

//

// . . . and a dozen others

//

Each synchronous lookup blocks waiting for a response before moving on to the next lookup.  The cumulative effect of these delays will get costly.  Ideally, you’d want to perform each lookup on its own thread and let the requests run concurrently.

So, let’s implement a method that looks like this:

public class AsyncDNSExample

{

public List<IPAddress> MultiHostLookup(List<string> hosts)

{

This method will accept a list of host names, execute concurrent DNS lookups for all of them, and return with a list of resolved addresses.  Here’s how we might use this method:

class Program

{

static void Main()

{

// build list of hosts to lookup

List<string> hosts = new List<string>();

hosts.Add(“www.gavaghan.org”);

hosts.Add(“www.itko.com”);

hosts.Add(“sombrita.com”);

// perform the concurrent lookup

AsyncDNSExample lookup = new AsyncDNSExample();

List<IPAddress> addressList = lookup.MultiHostLookup(hosts);

// write out the results

foreach (IPAddress address in addressList)

{

Console.WriteLine(address);

}

}

}

Begin and End methods

Many of .NET’s I/O classes have asynchronous versions of their synchronous methods.  For example, the Read() and Write() methods on System.IO.Stream have respective BeginRead() and BeginWrite() counterparts.  System.Net.Sockets.Socket has BeginAccept() and BeginConnect().  And, in benefit of this example, Dns has BeginGetHostAddresses().

All of the Begin* methods cause the object’s work to execute on a worker thread in the .NET thread pool.  These methods take the same parameters as their synchronous counterparts plus two additional parameters supporting the async framework.

For example, here’s the signature for BeginGetHostAddresses():

public static IAsyncResult BeginGetHostAddresses(

string hostNameOrAddress,

AsyncCallback requestCallback,

Object state

)

One added parameter is an AsyncCallback delegate.  The delegate identifies the callback method .NET will invoke once asynchronous processing has completed.  The callback method takes a single parameter of type IAsyncResult.  The IAsyncResult object must be used to access the result of the asynchronous call.

The second added parameter is an arbitrary state object (possibly null) that may be used to coordinate between the caller and the callback.  The state object is made available to the callback method through the IAsyncResult parameter.  An example is included a little later.

For each Begin* call, a corresponding End* call must be invoked to get the results of the method.  End* methods block synchronously until processing has been completed.  However, when called from within the callback method, End* methods return immediately because, at that point, the work is known to be done.

Let’s take a look at how our MultiHostLookup() method can be implemented using the asynchronous version of GetHostAddresses():

public class AsyncDNSExample

{

public List<IPAddress> MultiHostLookup(List<string> hosts)

{

// we’ll fill this list with the result of the DNS lookups

List<IPAddress> addressList = new List<IPAddress>();

foreach (string host in hosts)

{

// begin an asynchronous lookup for each host

Dns.BeginGetHostAddresses(

host,

GetHostAddressesCallback,

addressList

);

}

//

// we can do additional work here while the

// DNS lookups continue in parallel

//

lock (addressList)

{

// ensure all lookups have returned, otherwise wait

while (addressList.Count != hosts.Count)

{

Monitor.Wait(addressList);

}

}

return addressList;

}

}

This method begins by allocating the List<IPAddress> object we’ll use to return our resolved addresses.  Then, we loop over each of the host names in the hosts List<string> and call BeginGetHostAddresses().  Once this loop completes, all of the DNS queries are executing in parallel.

Notice the first parameter to BeginGetHostAddresses() is a host name – just like its synchronous counterpart.  For the second parameter, we pass a reference to our callback method, GetHostAddressesCallback(), which is defined below.  The third parameter is our result List<IPAddress>.  This will make the List available to the callback method.  When each DNS query completes, the callback method can update the list with each resolved address.

At this point, if we wanted to, we could code other logic to execute as we wait for the queries to complete.

Finally, we check the length of the list of resolved IP addresses to see if it’s the same length as the list of host names.  To do this, we must first lock the address list object (after all, we don’t want our address list modified on a callback thread at the same time we’re trying to inspect it).  If the list sizes are equal, we know all lookups have completed and we can return from the method.  Otherwise, we release the lock on the address list and block until receiving notification from the callback thread.

It’s all pretty simple.  Now, let’s see how to implement the callback method:

public class AsyncDNSExample

{

private void GetHostAddressesCallback(IAsyncResult result)

{

// This method may fail with a SocketException, particularly

// if the host is not found. A more robust solution would

// handle such cases.

IPAddress[] addresses = Dns.EndGetHostAddresses(result);

// for simplicity, we’ll take the first address

IPAddress address = addresses[0];

// the address list we passed in is accessbile from AsyncState

List<IPAddress> addressList = (List<IPAddress>)result.AsyncState;

// we need to ensure updates to the address list are threadsafe

lock (addressList)

{

addressList.Add(address);

// notify listeners that another address has been added

Monitor.PulseAll(addressList);

}

}

// this is our public method for performing multiple, concurrent

// DNS requests

public List<IPAddress> MultiHostLookup(List<string> hosts)

{

. . . .

}

}

The first thing that happens is the call to EndGetHostAddresses() with the IAsyncResult object passed in.  This call returns immediately with the result of the DNS query (it returns an array of IP addresses but, for simplicity, we’ll assume the first one is all we need).

Next, we get a reference to the List<IPAddress> object passed in by the Begin* call. This is where we’re going to save our resolved IP addresses.  However, for thread safety, we can only add our result from within a lock block.  We don’t want to be manipulating the list at the same time as another callback!

Finally, we pulse all threads listening on the result object.  This schedules the thread executing MultiHostLookup() to check if all of the results have been received.

Conclusion

Using asynchronous I/O can make your applications faster and your user interfaces more responsive – particularly when executing long running tasks and tasks that would otherwise leave the CPU idle. However, even with the .NET design pattern, multithreaded programming always adds to code complexity. So, only leverage this framework where performance optimization is required.

Developing software that uses SSL is an entirely different matter.  The simplicity quickly fades, and the developer must confront the complexities of certificate management, trust stores, handshaking, and a host of other details that must be perfectly aligned to make the secure communication work.  In Part 1, we’ll cover a very high level of SSL concepts.  In subsequent posts, we’ll take a deeper dive into making these connections happen in both Java and C#.

Understanding SSL

SSL uses public key/private key cryptography for three purposes. The most fundamental use is to encrypt data communication between the server and client.  However, it is also used to allow the server to prove its identity to the client and prevent man-in-the-middle attacks (where a malicious intermediary intercepts messages from the client and masquerades as the intended server).

A third use is for allowing the client to prove its identity to the server.  Mutual authentication is an important and powerful feature of SSL, and it’s probably underused.  For now, we’ll just focus on the semantics of server authentication.  If you understand server authentication, you’ll be well on your way to understanding client authentication on your own.

About Certificates

Three fundamental components are involved in setting up an SSL connection between a server and client: a certificate, a public key, and a private key.

Digital certificates are used to identify an entity.  The entity could be a person (when used for secure email), or it could be a computer (when used for SSL).  There is quite a bit of information stored in a digital certificate, but the most important part is the name of the entity it is identifying.

The identity, also known as the “subject”, is specified as an X.509 distinguished name.  A distinguished name contains multiple components.  For example, the distinguished name on the certificate used to setup an HTTPS connection for Amazon.com’s shopping cart check-out looks like this:

CN=www.amazon.com,

O=Amazon.com Inc.,

L=Seattle,

S=Washington,

C=US

How do we find this information?  From Internet Explorer, browse to any secure web page (one with an https:// protocol in the URL).  Right-click on the page and select “Properties”.  From the properties page, click the “Certificates” button.  On the “Details” tab, we can see all of the information embedded in the certificate.  By selecting the “Subject” entry, we can see the entity the certificate identifies:

For the purposes of establishing an SSL connection to a server, the only interesting part of the distinguished name is the “Common Name” specified by the “CN” component.  This is the name the server uses to identify the domain name of the host.

To establish a secure connection to Amazon.com, a client first resolves the domain name www.amazon.com.  After the SSL connection has been initiated, one of the first things the server will do is send its digital certificate.  The client will perform a number of validation steps before determining if it will continue with the connection.

Most importantly, the client will compare the domain name of the server it intended to connect to (in this case, www.amazon.com) with the common name (the “CN” field) found in the subject’s identity on the certificate.  If these names do not match, it means the client does not trust the identity of the server (and the client will likely choose to terminate the connection).

Although the server name may be correct, the client must still verify the integrity of the certificate to determine if it has been forged or tampered with.  The client does this by verifying the digital signature on the certificate.  We’ll talk more about digital signatures in a moment.

The client will also ensure the certificate is being used within a valid time frame.  All certificates contain an “issue date” and an “expiration date”.  A certificate is considered invalid outside of that date range.

Public and Private Keys

Public keys and private keys are number pairs with a special relationship.  Any data encrypted with one key can be decrypted with the other.  This is known as asymmetric encryption.  The security of asymmetric encryption lies in the difficulty of cracking encrypted data even when the key used for encryption is known.

The server’s public key is embedded within its certificate.  The public key is freely distributed so anyone wishing to establish an encrypted channel with the server may encrypt their data using the server’s public key.  The server will decrypt this message using its private key.  For this reason, private keys are closely guarded and kept secure.

Digital Signatures

Just as data may be encrypted with a public key and decrypted with a private key, the reverse is also true.  Data encrypted with a private key may be decrypted with the corresponding public key.

This property of keys is used to ensure the integrity of a digital certificate in a process called digital signing.

A hashing algorithm (such as SHA1 or MD5) is a means of processing all of the bytes of a message and producing a numeric “hash value”.  Hash values have long been used to ensure the integrity of messages that may become corrupted during transport.  A sender will transmit a message followed by the hash value it calculated for the message it intends to send.  The receiver calculates the hash value for the message it receives.  If the receiver calculates a different hash value than the one that was sent, the receiver concludes the message was corrupted during transit (and generally asks the sender to resend).

A hashing algorithm may also be used to determine if a message has been forged or tampered with.  Complicating this, however, is that a malicious third party could intercept a message, modify it, and simply recalculate the hash.  Asymmetric encryption technology solves this.

If a message sender wants to convince a recipient that his message is authentic and has not been tampered with, he will do two things.  First, the sender will calculate a hash value for the message.  This hash value will then be encrypted using the sender’s private key (a key which the sender, and only the sender, knows).  When the client receives the message, the client decrypts the hash value using the sender’s public key.  If the message has been tampered with, or if the message has been signed with anything other than the sender’s private key, the hash values will not agree and the client will not consider the message authentic.

Certificate Signing

When a certificate is created, it is digitally signed.  The digital signature is used to verify the authenticity of the certificate.  In an SSL connection, the client will attempt to verify the signature on the certificate presented by the server before deciding to continue establishing the connection.

Self-signed certificates

The simplest certificate is a self-signed certificate.  The signature on a self-signed certificate is calculated using the same private key associated with the public key found on the certificate.  In a software development environment, self-signed certificates are an easy way to build testing environments that establish SSL connections without having to deal with the time and expense of obtaining a certificate through an establish certificate authority.

Certificate Authority signed certificates

Certificates may also be signed by a certificate authority (CA).  A CA is a trusted third party that digitally signs certificates for entities that have gone through an established vetting process.  The CA, itself, also has a certificate that can be analyzed for authenticity – and the CA’s certificate might also be signed by yet another trusted third party (in this case, the CA is known as an intermediate CA).  All of these certificates, together, form a certificate chain.  At the top of the chain is a certificate authority called a Root CA that uses a self-signed certificate.

Returning to the Amazon.com example, we can examine the server certificate and see that it is not a self signed certificate – it is signed by a CA with a distinguished named of:

CN=VeriSign Class 3 Secure Server CA – G2,

OU=Terms of use at https://www.verisign.com/rpa (c)09,

OU=VeriSign Trust Network,

O=VeriSign, Inc.,

C=US

From the certificate details tab we looked at earlier, we can find the name of the certificate issuer by selecting the “Issuer” entry:

VeriSign is a trusted third party that issues certificates for, among other things, eCommerce applications.

Why is it important to have a certificate signed by a trusted third party?  It’s important because new HTTPS-based Web applications are being deployed all of the time.  In terms of browser-based, retail eCommerce applications, it’s simply impractical for users to manage a list of all of the server certificates they have decided to “trust”.  Furthermore, reliably and securely obtaining a web site’s real server certificate would be too problematic.

Consider a new Amazon.com customer.  When the shopping cart checkout sends Amazon.com’s certificate identifying itself as www.amazon.com, how does the customer know to trust the certificate?  Although the certificate may have a valid name and signature on it, how does the customer decide to trust the certificate?  If it is self-signed, it could have been signed by anyone – including a malicious man-in-the-middle.  What the customer needs is a reliable means of receiving Amazon’s certificate that is protected from forgery.

To do this, Amazon chose not to use a self-signed certificate.  Instead, for a fee, it requested that VeriSign sign its online shopping certificate.  When the customer receives Amazon.com’s certificate, she says “I trust that I have a legitimate copy of VeriSign’s CA certificate, I trust VeriSign to only sign certificates of the real domain name owners, and I can see that Amazon.com’s certificate is signed by VeriSign.  Therefore, I believe the server responding at www.amazon.com is truly owned and managed by the same entity owning the www.amazon.com domain name.”

Where Do Trusted CA’s Come From?

Why does the customer believe she has a legitimate copy of VeriSign’s CA certificate?  She believes this because a set of “trusted” CAs came pre-installed with her Web browser software.  Internet Explorer, Firefox, and all other leading browser vendors pre-configure their browsers to trust well known CAs such as VeriSign, Thawte, and Network Solutions.

C# programs will generally access the Windows Cryptographic Service Provider for trusted certificates.  The CSP is a shared OS resource usable by any program, and is also the same trust store consulted by Internet Explorer.

Similarly, the Java Runtime Environment comes with a pre-configure set of trusted certificate authorities.  The collection of trusted certificates can be found at [JRE_HOME]/lib/security/cacerts.  The keytool, a command line utility found in the SDK, can be used to inspect and manipulate this file.  The default password for the cacerts keystore is “changeit”.

Summary

Public key cryptography is at the heart of SSL.  While most developers are aware this technology is used to encrypt a data channel, most are unfamiliar with its use of digital signing for identity authentication and message validation.  It is the lack of understanding of these other uses that generally stymie their efforts to implement SSL.

Stayed tuned for more posts on the lower level details of implementing SSL technology,

One technology implemented by JAMES for spam filtering is real-time DNS blacklists.  DNSBLs identify the IP addresses of potential spam sources and machines known to be delivering spam (as determined by the sometimes controversial policies of the list owner).  Spam blacklists date back to 1996 with Paul Vixie’s Mail Abuse Prevention System, and are now used by ISPs and corporate mail systems around the world.  Countless organizations maintain blacklists, and Web sites like MX Toolbox permit ad hoc queries of IP addresses against dozens of published lists.

How It Works

Built around the UDP-based DNS protocol, a DNSBL is an efficient and lightweight mechanism for checking the IP addresses of incoming messages against a list of email senders a mail server may wish to avoid.  This is much like the concept of today’s Service Oriented Architectures – providing an uncoupled, standards-based interface consumed by arbitrary clients – except years ahead of its time when created.

Querying an IP address is as simple as reversing the octets of the address and appending the domain name of the list publisher.  Perform a DNS lookup of the “A record” for this string.  If a record is returned, the IP address is on the blacklist (some DNSBLs also return the reason for the listing in the TXT record).  If no record is found, the address isn’t listed.

Java, C#, and most other high level programming languages provide a means for performing a DNS lookup.  A simple way to try it out, however, is from a DOS prompt.  Suppose you want to check out the IP address 213.199.154.22 using the list maintained by 510 Software Group at blackholes.five-ten-sg.com.  You could use the nslookup command like this:

nslookup 22.154.199.213.blackholes.five-ten-sg.com

This should reply with “Non-existent domain”.  In other words, the address is “clean”.
To lookup the same address on the SpamCop blacklist at bl.spamcop.net, you would use this command:

nslookup 22.154.199.213.bl.spamcop.net

SpamCop also has a mechanism for simulating a positive response.  Technically, 127.0.0.2 is a local address.  But, SpamCop will provided a record for it:

nslookup 2.0.0.127.bl.spamcop.net

Try it out.  You’ll get back a valid “A record” indicating the address is listed – simulating the response you’d get for a blacklisted host.

How Addresses Get Listed

IP addresses get added to blacklists based on the policies selected by the list owners.  This is important to understand before blindly adding all available blacklists to your mail server.  Some lists are more aggressive than others, and the more aggressive a policy is the more likely you are to have legitimate email filtered out of your inbox.

Some spammers are identified by trusted sources forwarding spam messages to list managers.  The mail headers will identify the IP address of the sender.  Other spammers are identified when list owners plant honeypots – bogus email addresses posted online in order to identify spammers harvesting email address off of Web pages.

Some blacklists contain the IP addresses allocated to residential Internet subscribers regardless of whether they’ve been definitively identified as a spam source.  The rationale is that residential Internet users will use their ISP’s mail server to send and receive email.  Any email coming directly from a subscriber’s computer is either a deliberate spam campaign attempting to circumvent the ISP’s safeguards, or it’s a message generated by a zombie – a computer compromised by a virus and controlled by a hacker for the purpose of delivering spam.

Another category of blacklisted IP addresses belong to open relays – mail servers that don’t require authentication and thus provide a “hop” for spam messages to freely pass through.  Open relays allow spammers to hide the true origin of their messages (because the originating IP address might already be blacklisted).  This abuse of the open mail server often occurs without knowledge of the server owner.

All of these policies carry with them a bit of controversy.  A single spammer on a large network might cause thousands of innocent users on the same network to have their outbound email blocked.  Open relay owners are also regarded more as victims than active participants in spamming (although they should be summarily reprimanded for not applying the most basic of security measures to their mail exchange: SMTP authentication).  Also, when blacklists are used by ISPs, customers might unknowingly fail to receive wanted messages that were filtered out based on someone else’s definition of spam.

Available DNS Blacklists

DNSBLs are intended for use by mail service providers – not individual email system users.  If you administer a mail server, a comparison of available blacklists you may consider configuring can be found at the DNSBL Resource Stats Center.

JUnit Factory is often able to generate mock instances automatically for problematic classes. When automocking fails, the developer can improve coverage by either extracting behaviors into private methods or by providing hints to JUnit Factory in the form of test data helpers.

This post is part of a series:

1. Characterization Tests: How To Deal With Legacy Java Code
2. JUnit Factory Part 1: Generating Tests
3. JUnit Factory Part 2: Finding Regressions
4. JUnit Factory Part 3: Improving Code Coverage

Handling External Resources

A classic unit testing problem is testing business logic that depends on an external resource. The external resource is usually a database, but it could also be a Web service, the file system, or a user interface. This is challenging because a true unit test executes in the absence of external dependencies, but creating mocks is an expensive, laborious process for the developer.

JUnit Factory’s answer to this is the Mockingbird framework. When code to be tested must retrieve data from a database, JUnit Factory will repeatedly see some failure (like a SQLException) being thrown every time it tries to execute a database call. To move beyond the database call, JUnit Factory will create a mock implementation of the failing method.

Let’s look at a modified version of the makePurchase() method we discussed earlier:

public void makePurchase(Purchase purchase) throws CreditCardException

{

validateBalance(balance + purchase.getAmount());

CreditCardDAO dao = CreditCardDAO.getDAOImplementation();

dao.recordPurchase(accountNumber, purchase);

balance = balance + purchase.getAmount();

}

We’ve introduced a data access object called CreditCardDAO encapsulating our interface to an external system. In this case, it’s probably a database. There’s likely one or more subtypes of CreditCardDAO, such as CreditCardOracleDAO or CreditCardMySqlDAO, providing implementation-specific behaviors for the data storage system.

Notice there are no subtypes of CreditCardDAO in the example Eclipse project archive. They wouldn’t be usable, anyway, since a properly written unit test should execute in the absence of a database. A developer would need to create a mock instance of CreditCardDAO in order to write the test.

If we submit this class to JUnit Factory to autogenerate a characterization test, we can see how the call to CreditCardDAO.recordPurchase() gets mocked out using the Mockingbird framework:

public void testMakePurchaseWithAggressiveMocks() throws Throwable

{

CreditCard creditCard =

(CreditCard) Mockingbird.getProxyObject(CreditCard.class, true);

Purchase purchase =

(Purchase) Mockingbird.getProxyObject(Purchase.class);

CreditCardDAO creditCardDAO = new CreditCardDAO();

setPrivateField(creditCard, “creditLimit”, new Double(0.0));

setPrivateField(creditCard, “accountNumber”, “”);

setPrivateField(creditCard, “balance”, new Double(0.0));

setPrivateField(purchase, “amount”, new Double(0.0));

CreditCardDAO.setDAOImplementation(creditCardDAO);

Mockingbird.enterRecordingMode();

Mockingbird.setReturnValue(false,

creditCardDAO,

“recordPurchase”,

“(java.lang.String,example2.Purchase)void”,

null,

1

);

Mockingbird.enterTestMode(CreditCard.class);

creditCard.makePurchase(purchase);

assertEquals(“creditCard.getBalance()”,

0.0,

creditCard.getBalance(),

1.0E-6

);

}

First, notice the suffix “WithAggressiveMocks” added to the end of the test method name. This indicates JUnit Factory needed to create mocks for a few classes in order to create a test. Aggressive mocks aren’t a problem, although they can make tests harder to read. Later, we’ll discuss test data helpers which can sometimes be used to avoid aggressive mocking and create clearer test code.

The test method begins with Mockingbird creating a number of proxy instances to create a hook into the call to be mocked. The actual insertion of the mock behavior occurs on the line that begins “Mockingbird.setReturnValue“.

Instead of instantiating a subtype of CreditCardDAO, JUnit Factory simply uses the base type. However, the base implementation of recordPurchase() always throws a RuntimeException indicating it must be overridden by a subclass. The “setReturnValue()” call adjusts the behavior of recordPurchase() to return normally instead. This allows the rest of the makePurchase() method to execute and update the balance.

These are unit tests – not integration tests

You’ll see Mockingbird used in a number of scenarios, not just where calls to external resources would otherwise fail. Mockingbird is applied anywhere JUnit Factory needs to control the return value of dependent types in order to continue executing a method.

This may seem a little disconcerting at first. After all, how good are these tests if we’ve mocked the behavior of dependent classes? This is okay. Remember these are characterization tests. This isn’t about asserting the correct behavior of our code. We’re only trying to capture the behavior of legacy code so we can find regressions.

Method extraction and test data helpers

Next, let’s take a look at a method called fraudCheck(). This implements an admittedly simplistic rule that flags account activity as suspicious if there are several small purchases (less than $10) that add up to over $100.

public boolean fraudCheck()

{

CreditCardDAO dao = CreditCardDAO.getDAOImplementation();

List purchases = dao.getRecentPurchases(accountNumber);

boolean isSuspicious;

double purchaseTotal = 0;

for (Iterator iter = purchases.iterator(); iter.hasNext();)

{

Purchase purchase = (Purchase) iter.next();

if (purchase.getAmount() < 10.00)

{

purchaseTotal += purchase.getAmount();

}

}

if (purchaseTotal > 100.00)

{

isSuspicious = true;

}

else

{

isSuspicious = false;

}

return isSuspicious;

}

When we execute CreditCardAgitaTest, we can look at the coverage information provided by the JUnit Factory plug-in on the left side of the editor window:

Here, we see line 138 is never executed when we run the test. For a series of purchases to be flagged as suspicious, we’d need at least 11 transactions of $9.99. Although JUnit Factory can mock the return value of getRecentPurchases(), it was unable to discover the proper state of a purchase list that would result in the execution of this line. So, we’re not able to reach all branches of our code.

Controlling preconditions

We know how to create a list of purchases that satisfies the criteria for suspicious activity, but we need some way of inserting that list into a test.

A unit test involves creating a set of preconditions to a method (the parameters) and asserting a set of postconditions (the state manipulated by the method). Before we can control the list of purchases as a test precondition, we need to refactor our code to turn the list of purchases into a method parameter.

The safest way to do this is to use Eclipse’s refactoring tools. Select the code in fraudCheck() from the first line after the call to getRecentPurchases() and on to the end of the method. Right-click the highlighted code in the editor and select “Refactor -> Extract Method”. We’ll name the new method “hasSuspiciousActivity()“.

This is completely safe because extracting this code into its own method doesn’t actually change the behavior of the fraudCheck() method. All we’ve done is make the fraudCheck() method testable. The new code looks like this:

public boolean fraudCheck()

{

CreditCardDAO dao = CreditCardDAO.getDAOImplementation();

List purchases = dao.getRecentPurchases(accountNumber);

return hasSuspiciousActivity(purchases);

}

private boolean hasSuspiciousActivity(List purchases)

{

boolean isSuspicious;

double purchaseTotal = 0;

for (Iterator iter = purchases.iterator(); iter.hasNext();)

{

Purchase purchase = (Purchase) iter.next();

if (purchase.getAmount() < 10.00)

{

purchaseTotal += purchase.getAmount();

}

}

if (purchaseTotal > 100.00)

{

isSuspicious = true;

}

else

{

isSuspicious = false;

}

return isSuspicious;

}

What we’ve achieved is the creation of a method that takes a list of purchases as a parameter. Now, we can give JUnit Factory a hint about how to create this parameter in such a way that isSuspicious will sometimes be set to “true”.

Giving hints using test data helpers

A test data helper is simply a class that creates objects JUnit Factory can pass as parameters to methods being tested. Test data helper methods must 1) have a name that begins “create”, 2) take no parameters, and 3) return an instance of the target data type. You may have as many test data helper methods on a test helper class as you want.

In the sample Eclipse project archive, look for a source folder named “testhelpers”. In there you’ll find a class named PurchaseListTestHelper. This class implements com.agitar.lib.TestHelper, so JUnit Factory will recognize it as a global test helper (as opposed to the ScopedTestHelper interface which ties the helper methods to a particular type). The class contains a single test data helper method named createSuspiciousPurchaseList(). Here is the complete class definition:

package example2;

import java.util.*;

import com.agitar.lib.TestHelper;

public class PurchaseListTestHelper implements TestHelper

{

public static List createSuspiciousPurchaseList()

{

ArrayList purchaseList = new ArrayList();

try

{

for (int i = 0; i < 11; i++)

{

Date purchaseDate = new Date(1000);

double amount = 9.99;

Purchase purchase = new Purchase( purchaseDate, amount );

purchaseList.add(purchase);

}

}

catch( CreditCardException exc )

{

throw new RuntimeException(“Failed to create purchase list”);

}

return purchaseList;

}

}

This helper create a list of 11 purchases of $9.99 each. If this object is passed to our hasSuspiciousActivity() method, we can expect it to execute the line setting isSuspicious to true.

Can we be sure JUnit Factory will use our test helper? Not necessarily. JUnit Factory prefers to use developer provided test data helpers whenever possible – as long as they provide unique code coverage or an interesting outcome. If you have multiple test helpers, but they all provide the same test coverage, JUnit Factory will only use one of them.

If no test helper provides a desired execution flow, JUnit Factory will create an instance of an object on its own by reflecting constructors off of the Class object. It is only when this final strategy fails that JUnit Factory resorts to aggressive mocks and the Mockingbird framework.

Will JUnit Factory use the test data helper we just created? We can generate tests for CreditCard again and find out. JUnit Factory will search our entire Eclipse project for all available test data helpers. There’s nothing to configure, because all classes implementing TestHelper or ScopedTestHelper will be used as candidates.

After regenerating tests for CreditCard, we can see the answer is “yes,” JUnit Factory found our test data helper and used it. Some of the test cases for hasSuspiciousActivity() use PurchaseListTestHelper as shown in the generated test:

public void testHasSuspiciousActivity() throws Throwable

{

CreditCard creditCard = new CreditCard(“3317 3013 6259 0300″, 100.0, 0.0);

List suspiciousPurchaseList =

PurchaseListTestHelper.createSuspiciousPurchaseList();

boolean add =

suspiciousPurchaseList.add(new Purchase(new Date(100L), 100.0));

boolean result = ((Boolean) callPrivateMethod( “example2.CreditCard”,

“hasSuspiciousActivity”,

new Class[] {List.class},

creditCard,

new Object[] {suspiciousPurchaseList})

).booleanValue();

assertTrue(“result”, result);

}

When we execute CreditCardAgitarTest, the code coverage bars in the editor window show JUnit Factory is now covering all of our business logic in CreditCard. It has also created additional assertions in CreditCardAgitarTest to check the outcome when isSuspicious returns true.

Test coverage strategy

With so many tools at your your disposal, where do you begin?

First, don’t concern yourself with aggressive mocks. Aggressive mocks happen automatically, and they’re okay. They’re merely a sign your legacy code wasn’t written for testability. Aggressive mocks give you the code coverage you need to detect changes in the behavior of your code. If you have 100% code coverage already, the only reason you might look into one of the other two strategies is if you want to improve the readability of your tests.

If aggressive mocks don’t give you enough coverage, consider refactoring your code using the Eclipse tool to extract a method. This allows you to create method parameters out of local variables that JUnit Factory couldn’t get into an appropriate state. Often times, you won’t even need to create a test data helper after refactoring your code. If the required state for a parameter is straightforward enough, JUnit Factory can figure out on its own how to create it properly.

Reach for test data helpers last. They’re a powerful tool. But, they require writing additional code – and that’s just more code you’ll need to maintain. Use them only when refactoring doesn’t give you the coverage you need, or when team development policies preclude modifying existing code.

Lastly, always consider whether the missing coverage is worth solving. You might be dealing with truly dead code that can never be reached. You might also be dealing with one or two minor lines that simply aren’t worth the effort. Aiming for 100% code coverage is a costly and, generally, foolish objective. Consider the cost against the benefit.

After all, if you’ve brought your test coverage from 0% up to 80% with the single click of a button, aren’t you profoundly better off than before?

This post is part of a series:

1. Characterization Tests: How To Deal With Legacy Java Code
2. JUnit Factory Part 1: Generating Tests
3. JUnit Factory Part 2: Finding Regressions
4. JUnit Factory Part 3: Improving Code Coverage

A simple requirements change

Suppose we need to implement a simple requirements change to the CreditCard class we’ve already created. Presently, the validateBalance() method disallows negative balances. The business leadership of our company, however, has decided to allow customers to overpay their credit card balances.

Before writing any code, we must first verify that all of our current characterization tests still pass. The fastest way to do this is to select the source folder containing our characterization tests in the Eclipse Package Explorer. Right click on the folder and select “Run As / Agitar JUnit Test”.

If any tests fail, it means some other developer has already introduced a behavior change into the system without addressing how this affects tests. If all of the tests pass, we know we have a safety net. We may now move forward making changes confident any regressions will be detected before we commit our work to version control.

Credit card payments are recorded in the makePayment() method:

public void makePayment( double amount ) throws CreditCardException

{

if (amount <= 0)

{

throw new CreditCardException(“Payment amount must be positive”);

}

validateBalance( balance – amount );

balance = balance – amount;

}

After ensuring a postive payment amount has been made, the method invokes validateBalance() on the proposed new balance to check if it’s legal.

Presently, validateBalance() disallows negative values, but we’ll comment out the section of the method that performs this check:

private void validateBalance( double balance ) throws CreditCardException

{

//if (balance < 0.00)

//{

// throw new CreditCardException(“Balance can’t go below minimum balance”);

//}

if (balance > creditLimit)

{

throw new CreditCardException(“Balance can’t exceed credit limit”);

}

}

The makePurchase() method now allows payments that exceed the outstanding balance on the credit card.

Rerun the tests to find behavior changes

When we rerun our characterization tests, we discover that three of the tests are now failing. It shouldn’t come as any surprise, though, since changing the behavior of the class is what we intended to do.

So, we should just regenerate the CreditCard tests, right? Absolutely not! We must first analyze each of the test failures to determine if all of the failures reflect an expected change in behavior.

One failing test is for validateBalance():

public void testValidateBalanceThrowsCreditCardException() throws Throwable

{

CreditCard creditCard = new CreditCard(“2298 9812 4566 1184″, 100.0, 0.0);

try

{

callPrivateMethod(“example1.CreditCard”,

“validateBalance”,

new Class[] {double.class},

creditCard,

new Object[] {new Double(-1.0)}

);

fail(“Expected CreditCardException to be thrown”);

}

catch (CreditCardException ex)

{

assertEquals(“ex.getMessage()”, “Balance can’t go below minimum balance”, ex.getMessage());

assertThrownBy(CreditCard.class, ex);

}

}

This test passes -1.0 to validateBalance() and expects a CreditCardException to be thrown. This test passed when run against the old code, but it fails now because an exception is no longer thrown for negative balances. This is an expected test failure. The failing test reflects an intended change of behavior.

As you might expect, there’s also a failing test for makePayment():

public void testMakePaymentThrowsCreditCardException() throws Throwable

{

CreditCard creditCard = new CreditCard(“2298 9812 4566 1184″, 1000.0, 100.0);

creditCard.makePayment(0.10000000149011612);

creditCard.makePurchase(new Purchase(new Date(0L), 1.4178674221038818));

creditCard.makePurchase(new Purchase(new Date(100L), 100.0));

creditCard.makePurchase(new Purchase(new Date(1000L), 3.9999998989515E-5));

creditCard.makePurchase(new Purchase(new Date(1L), 1.0E-5));

creditCard.makePurchase(new Purchase(new Date(-1L), 3.6909053325653076));

creditCard.makePayment(100.0);

creditCard.makePayment(5.630099296569824);

creditCard.makePayment(1.8264453411102295);

creditCard.makePayment(97.55227811549801);

try

{

creditCard.makePayment(0.0010);

fail(“Expected CreditCardException to be thrown”);

}

catch (CreditCardException ex)

{

assertEquals(“ex.getMessage()”,

“Balance can’t go below minimum balance”,

ex.getMessage());

assertThrownBy(CreditCard.class, ex);

assertEquals(“creditCard.getBalance()”,

0.0,

creditCard.getBalance(),

1.0E-6);

}

}

This is somewhat predictable since makePayment() depends on validateBalance(). This tests performs a series of purchase and payment transactions and checks that a payment creating a negative balance throws an exception. We no longer want such transactions to throw an exception, so this, too, is an expected test failure.

The third test failure, however, is cause for concern. This test of the constructor fails:

public void testConstructorThrowsCreditCardException1() throws Throwable

{

try

{

new CreditCard(“2298 9812 4566 1184″, 15000.0, -1.0);

fail(“Expected CreditCardException to be thrown”);

}

catch (CreditCardException ex)

{

assertEquals(“ex.getMessage()”,

“Balance can’t go below minimum balance”,

ex.getMessage()

);

assertThrownBy(CreditCard.class, ex);

}

}

This test checks that an exception is thrown when creating a CreditCard with a negative opening balance. Our requirements change permits a balance to go negative when making a payment, but not when opening a new account. However, we inadvertently changed the behavior of the constructor because the constructor, just like the makePayment() method, is dependent on validateBalance()!

Real regressions are usually far more obscure

This is a rather simplistic example of where a change in one part our code creates a regression somewhere else. In our CreditCard class, it would be trivial for a developer to grasp the entire behavior of the class and foresee the impact to the behavior of the constructor.

In real world applications, however, large legacy code bases contain complex relationships spanning hundreds of classes. It’s simply not possible for a developer to approach unfamiliar code and understand all of the interdependencies. Characterization tests will highlight behavior changes introduced by the developer, and allow the developer to analyzes those changes for correctness.

So, what shall we do about the regression we introduced in CreditCard? One solution is to create a new method for validating the opening balance. Invoke that method in the constructor instead of validateBalance():

public CreditCard( String accountNumber, double creditLimit,

double balanceTransfer ) throws CreditCardException

{

// … validate and assign account number and credit limit

// validate the balance

//validateBalance( balanceTransfer );

validateOpeningBalance(balanceTransfer);

this.balance = balanceTransfer;

}

private void validateOpeningBalance( double balance )

throws CreditCardException

{

if (balance < 0.00)

{

throw new CreditCardException(“Opening balance can’t be negative”);

}

if (balance > creditLimit)

{

throw new CreditCardException(“Balance can’t exceed credit limit”);

}

}

When we run our characterization tests now, the constructor test passes and the only failing tests are the expected failures.

Regenerate tests and commit changes

Now that we’re convinced we haven’t introduced any behavior changes into the code except for the changes we meant to implement, we can commit the updated CreditCard class to version control.

At the same time, we also should regenerate our CreditCard characterization tests and commit the new tests to version control. The update characterization tests will reflect the new behavior, and they’ll be available the next time a requirements change is needed.

In my next post, we’ll start to look at code coverage issues. Sometimes, JUnit Factory is unable to figure out how to execute all paths of your code. This could be due to the need for objects to exist in a complex state, the need to interact with an external resource such as a database, or simply due to dead code. Look for this post in the next few weeks.

This article describes how to generate tests for a simple Java class and how to read the tests. Not all of your real code will be this simple, and not all generated tests will be this simple, either. But, bear with me as we start small and work our way up.

This post is part of a series:

1. Characterization Tests: How To Deal With Legacy Java Code
2. JUnit Factory Part 1: Generating Tests
3. JUnit Factory Part 2: Finding Regressions
4. JUnit Factory Part 3: Improving Code Coverage

Let’s create a simple class

If you want to follow along, you can download the characterization tests sample code as an Eclipse project archive.

The sample code defines a simplistic CreditCard type. It’s admittedly inappropriate for real world use, but it’ll make for excellent illustration. The complete class definition can be found in the sample code download. For brevity, let’s start by looking just at the constructor:

public class CreditCard

{

private String accountNumber;

private double creditLimit;

private double balance;

/**

* Construct a new CreditCard

* @throws CreditCardException if any parameter is invalid

*/

public CreditCard( String accountNumber, double creditLimit,

double balanceTransfer ) throws CreditCardException

{

// validate the account number

if (!accountNumber.matches(“(\\d{4}[ ]){3}\\d{4}”))

{

throw new CreditCardException(“Invalid credit card number”);

}

this.accountNumber = accountNumber;

// validate the credit limit

if (creditLimit <= 0)

{

throw new CreditCardException(“Credit limit must be positive”);

}

if (creditLimit > 15000)

{

throw new CreditCardException(“Credit limit may not exceed 15,000″);

}

this.creditLimit = creditLimit;

// validate the balance

validateBalance( balanceTransfer );

this.balance = balanceTransfer;

}

public double getBalance()

{

return balance;

}

public double getCreditLimit()

{

return creditLimit;

}

//

// …other methods omitted for clarity

//

}

The constructor is pretty straightforward. It accepts an account number, a credit limit, and a starting balance (our credit card company allows new accounts to be created with a transferred opening balance).

The behavior of our constructor is to validate all three of these parameters. The account number must be of the form “nnnn nnnn nnnn nnnn”. The credit limit must be positive, and our cautious credit card company never grants a credit limit greater than $15,000. Finally, the opening balance is validated by a method called validateBalance() which we’ll look at later.

We’ll generate characterization tests for this class by selecting the class in the Eclipse package explorer and clicking the test generation button on the toolbar as shown in the screenshot below:

This will send our project to JUnit Factory which will analyze and execute our code, make observations about its behavior, and generate a series of unit tests which will capture those observations. This will take several seconds.

The generated tests

If you’re already familiar with JUnit, you know JUnit test cases for a class under test will extend junit.framework.TestCase. JUnit Factory extends this type with AgitarTestCase. This is the base class of all JUnit Factory generated tests. So, our CreditCard test case is declared like this:

import com.agitar.lib.junit.AgitarTestCase;

public class CreditCardAgitarTest extends AgitarTestCase

{

//

// …generated tests

//

}

AgitarTestCase provides additional data gathering and utility methods used by the tests. You don’t need to worry about adding the Agitar jar files to your classpath because the Eclipse plug-in will automatically configure your project the first time you send a test generation request.

If you’re following along, the tests you get back might look slightly different than what we’re about to look at – particularly if you’re typing it in by hand. Tests will vary based on the presence (or absence) of property getters and setters, other methods which might produce execution coverage, and the latest test generation techniques developed by Agitar Labs.

You can expect to get back several tests for the constructor. One positive test case looks like this:

public void testConstructor1() throws Throwable {

CreditCard creditCard = new CreditCard(“2298 9812 4566 1184″,

13.872842788696289,

13.872842788696289);

assertEquals(“creditCard.accountNumber”,

“2298 9812 4566 1184″,

getPrivateField(creditCard, “accountNumber”));

assertEquals(“creditCard.getCreditLimit()”,

13.872842788696289,

creditCard.getCreditLimit(),

1.0E-6);

assertEquals(“creditCard.getBalance()”,

13.872842788696289,

creditCard.getBalance(),

1.0E-6);

}

This test starts by instantiating a CreditCard with legal values (we’ll cover negative test cases later). Notice JUnit Factory figured out how to create credit card numbers in the required format. How is this possible? When analyzing the bytecode, JUnit Factory found this line:

if (!accountNumber.matches(“(\\d{4}[ ]){3}\\d{4}”))

Because regex was used validate the account number string, JUnit Factory used this information to create a legal parameter.

Arbitrary values appear for the credit limit and starting balance. These values, of course, have far too many digits after the decimal point to be legitimate monetary values. This is because there’s nothing in the code to indicate there should be only two digits after the decimal point. JUnit Factory simply chose values that allowed the constructor to execute without throwing an exception.

Following the constructor are a series of assertions reflecting the state of the CreditCard object observed by JUnit Factory. First, JUnit Factory observed the private attribute accountNumber was set to the same value as the accountNumber parameter. Notice this assertion uses reflection because accountNumber is a private attribute. There is no public getter for accountNumber.

The assertions for the credit limit and balance are similar, but notice JUnitFactory discovered JavaBeans-style getters for these attributes and uses them instead of reflection. This creates a test that is more readable. This also creates a test that is sensitive to changes in both the class’s constructor and some of its method members.

Here’s another interesting test generated for the constructor:

public void testConstructor2() throws Throwable {

CreditCard creditCard = new CreditCard(“2298 9812 4566 1184″,

15000.0,

0.0010);

assertEquals(“creditCard.accountNumber”,

“2298 9812 4566 1184″,

getPrivateField(creditCard,

“accountNumber”));

assertEquals(“creditCard.getCreditLimit()”,

15000.0,

creditCard.getCreditLimit(),

1.0E-6);

assertEquals(“creditCard.getBalance()”,

0.0010, creditCard.getBalance(),

1.0E-6);

}

This test looks similar to the previous test, but note that 15000.0 is passed for the credit limit. This time, the credit limit value is not arbitrary. By analyzing the constructor bytecode, JUnit Factory noticed that 15000.0 is an interesting value – a branch of execution is triggered based off this corner condition. Thus, JUnit Factory generates a test for this scenario.

JUnit Factory also generates negative test cases – tests that capture the behavior of a method when things go wrong. Here’s one such test generated to ensure an exception is thrown when the credit limit parameter is less than zero:

public void testConstructorThrowsCreditCardException3() throws Throwable {

try

{

new CreditCard(“2298 9812 4566 1184″, -0.0010, 100.0);

fail(“Expected CreditCardException to be thrown”);

}

catch (CreditCardException ex)

{

assertEquals(“ex.getMessage()”,

“Credit limit must be positive”,

ex.getMessage());

assertThrownBy(CreditCard.class, ex);

}

}

How many tests does JUnit Factory generate? In this case, four tests for normal outcomes, and sevens tests for exceptional outcomes. JUnit Factory attempts to generate just enough tests for 100% code coverage. You might wonder why there aren’t tests using a richer data set with hundreds of possible input combinations. The answer is simple: those tests don’t add value. There are no other input parameters that would provide additional code coverage.

Unit testing private methods

JUnit Factory easily supports testing of private methods by invoking those methods through reflection.

The second to last line of the CreditCard constructor invokes validateBalance() to test whether the opening balance is legal. Here’s how validateBalance() is implemented:

private void validateBalance( double balance ) throws CreditCardException

{

if (balance < 0.00)

{

throw new CreditCardException(“Balance can’t go below minimum balance”);

}

if (balance > creditLimit)

{

throw new CreditCardException(“Balance can’t exceed credit limit”);

}

}

This method tests whether a proposed balance value falls within a range. The behavior has been extracted to a private method because it is shared by multiple methods. JUnit Factory is able to independently test this behavior using reflection. Four such tests, two positive and two negative, are generated for validateBalance(). Here’s one example:

public void testValidateBalance() throws Throwable {

CreditCard creditCard = new CreditCard(“2298 9812 4566 1184″,

100.0, 0.0);

callPrivateMethod(“example1.CreditCard”,

“validateBalance”,

new Class[] {double.class},

creditCard,

new Object[] {new Double(0.0)}

);

assertEquals(“creditCard.getCreditLimit()”,

100.0,

creditCard.getCreditLimit(),

1.0E-6);

}

This test creates a CreditCard object and invokes validateBalance() by passing 0.0. Afterward, the test asserts that the validation method, which utilizes the creditLimit attribute, leaves the credit limit value unchanged.

This test is harder to read, but it is this sort of complexity that makes unit testing private methods costly and practically nonexistent in tests created by hand. With JUnit Factory, the test is written with the click of a button.

This feature adds tremendous value. Independently testing private methods improves a developer’s ability to track down the source of behavioral changes whenever tests fail.

This post is continued:
JUnit Factory Part 1: Generating Tests (page 2) »

As a result, Java developers seldom have the luxury of working on true greenfield projects. Instead, they are faced with adding enhancements and fixing bugs on projects built upon a code base they didn’t write and don’t fully understand. How can developers safely make changes to legacy code without accidentally breaking something unrelated?

Characterization tests provide a safety net – a change detection engine – that identifies behavioral changes in legacy code in order to remedy regressions early in the development process. Fixing regressions early shortens development timelines, increases code quality, and allows a team to become more agile. You can automatically generate your team’s characterization tests using the free JUnit Factory for Java from Agitar Software.

This post is part of a series:

1. Characterization Tests: How To Deal With Legacy Java Code
2. JUnit Factory Part 1: Generating Tests
3. JUnit Factory Part 2: Finding Regressions
4. JUnit Factory Part 3: Improving Code Coverage

Use JUnit Tests for Regression Testing – If You’re Lucky Enough to Have Any

If you’re lucky, you inherit legacy Java code that already comes with a full JUnit test suite. Running unit tests written by the previous development team can detect unintended behavior changes introduced by new modifications. Running these tests on a regular basis can identify regressions long before they reach QA (where they’re more costly to fix).

The reality of existing unit tests, however, is they generally cover only a small subset of the behavior of the code – core business logic or especially complex algorithms. There is seldom substantial coverage of the entire code base, and there is even less focus on negative test cases (tests that assert a particular response when things go wrong).

What’s even worse is, more often than not, unit tests are nonexistent. Is this because developers are lazy? Maybe. The more likely explanation, however, is that maintaining unit tests is hard! It is both time consuming and expensive.

Unit testing is a combinatorial problem – every boolean decision statement requires at least two tests: one with an outcome of “true” and one with an outcome of “false”. Developers need to write 3 to 5 lines of test code for every line of production code to be tested. It is enough work to write tests in the first place, but refactoring the application code requires rewriting the corresponding unit tests, too.

Developers aren’t punished for bugs – they’re punished for missing deadlines. Consequently, when given a choice between delivering on time or delivering quality software, software quality – along with the unit tests intended to ensure it – gets ignored at crunch time.

Characterization Tests are not Functional Tests

If you’ve ever written a unit test, you probably wrote it to assert particular behaviors in the code you just wrote. If you’re in a Test-Driven Development environment, you wrote the tests first, and then you went on to write just enough code to get your tests to pass.

Characterization tests are different. The term was coined by Michael Feathers in his book “Working Effectively with Legacy Code”. Characterization tests are usually generated using an automated tool like the free JUnit Factory for Java from Agitar Software. Rather than asserting the correctness of a code unit, characterization tests simply capture the behavior of the code, as written, in order to detect behavioral changes later.

Obtaining meaningful code coverage from hand written JUnit tests is difficult, and aiming for any substantial coverage target is generally not cost effective. Remember, you’ll be writing 3 to 5 times more Java code if you’re trying to achieve 100% coverage. Automatically generating characterization tests using JUnit Factory, however, can give you 80% coverage or better with just the click of a button.

How to Use Characterization Tests

The JUnit tests generated by JUnit Factory simply capture how your legacy code behaves today. You’ll want to generate tests for your entire legacy code base before you begin new work. This will create a comprehensive test suite, providing at least 80% coverage, that will detect behavioral changes as you implement new functionality.

With this safety net in place, you can write new code with confidence. But, what does it mean when your tests fail?

A failed characterization test doesn’t always mean a bug. In fact, if you’re intentionally adding new functionality, you should expect some of your tests to fail. After all, altering the behavior of your code was the goal you set out to achieve.

However, be sure you can explain the failure of every test as an intended change. If a unit test fails that you wouldn’t expect to fail, it means you’ve introduced a regression. Continue working on your code until those tests pass.

Distinguishing between expected and unexpected characterization test failures is where the analytical skills of the developer come into play. Work is complete when the only failing tests are the ones expected to fail. At that point, you may safely commit your changes to version control with a high level of confidence you haven’t broken anything.

You’ll also want to use JUnit Factory to generate a new set of tests for the Java classes you modified. This will capture the behavior of your new code, identify future regressions, and eliminate the tests that failed due to deliberate changes. After all, today’s greenfield projects are tomorrow’s legacy code!

How Does JUnit Factory Do It?

JUnit Factory is far more sophisticated than a static analysis tool. In addition to analyzing, and attempting to reach, all of the boundary conditions and branches in your code, JUnit Factory will actually execute your code – instantiating complex parameter objects by reflecting constructors – and record outcome states and exceptions.

What about code that requires data from some external source like a database or a Web service? JUnit Factory is surprisingly effective at auto-mocking external resources using its Mockingbird framework. External dependencies are eliminated to create repeatable unit tests that are independent of their environment.

How To Get Started

Download the free Eclipse plugin for JUnit Factory and try it out on some of the classes in your own projects. Inspect the tests it generates, and you’ll get a feel for what’s going on. Unresolved dependencies get mocked out, private methods get called by reflection, and both positive and negative tests are created.

The Eclipse plugin will also report on the code coverage achieved by the generated tests. Sometimes, certain fragments of code require classes to enter a complex state in order to get executed. If JUnit Factory can’t figure out how to create this precondition, you’ll have some untested sections of code. However, JUnit Factory can, if necessary, be configured with test data helpers that provide hints on how to create this coverage and improve the quality of your tests.

Is this the end of functional testing? No. It’s still a good idea to create functional tests to assert the behavior of your newly written code. After all, that’s the only way to verify intended behavior. Use your hand coded tests as a complement to JUnit Factory change detection tests, and see them all reported together in JUnit Factory’s testing dashboard.

Legacy code is often ugly, but it isn’t going away – companies have invested too many resources to scrap it and start over. Characterization tests make ugly legacy code maintainable, and JUnit Factory produces characterization tests with low cost, high coverage, and a lack of drudgery for which all developers will be grateful.

Go to “JUnit Factory Part 1: Generating Tests” »