Visual Studio Toolbox

HTTP Debugging Tools: Buy, Borrow or Build Your Own Tools?

When you need a specialty development tool like an HTTP debugger, you have a few options. I'll take a look at the popular current offerings and talk to some of their developers about the experience of building these apps.

• By Terrence Dorsey
• 07/14/2014

The topic I write most about here at VisualStudioMagazine.com is handy tools and learning resources for software developers. Usually free tools and resources, at that.

Occasionally, I come across developers complaining, "Why hasn't someone built an app to do [insert urgent need here]?" and I wonder: These are developers, so why don't they build a solution themselves?

Of course, that's a deeply unfair thought. We all have work that needs to get wrapped up ... usually sooner than we'd like. And we're not experts at every aspect of the technologies and programming languages we encounter. It isn't realistic to think we'll all just sit down and start yak shaving up a tool every time we want to get our work done better or faster.

Why, we'd never get the work done at all with such distractions.

And yet, smart folks out there do build the tools that help us get our work done. Instead of just wondering who these folks are and why they build their applications, I went out to find a few of them and ask -- specifically in the area of HTTP packet analyzers. I ended up getting the chance to chat with the developers of two of these apps – Charles and Fiddler -- to find out more about why they created them.

HTTP packet analyzers also happen to fall across the spectrum from free and open source to free and closed source to paid apps at various price points and levels of complexity. This is one area where the tools are pretty sophisticated and crucial to debugging Web-enabled apps. In addition to talking with the developers of Charles and Fiddler, I took the opportunity to survey that larger spectrum of free and paid offerings.

So here's a look at the HTTP debugging landscape. I hope you find it interesting and helpful.

Wireshark
I'll start with Wireshark. It's a free, open source network analysis tool that lets you capture and inspect not only HTTP traffic, but a seemingly endless variety of other wired and wireless network communication protocols.

There are versions for Windows, Mac, Linux and several other *nix platforms, which makes it a handy tool to learn if you're engaged in cross-platform development.

How-To Geek has an in-depth tutorial, " How to Use Wireshark to Capture, Filter and Inspect Packets," that will get you started, and Himanshu Aurora at The Geek Stuff compiled a handy collection of examples, " Wireshark Display Filter Examples (Filter by Port, IP, Protocol)," as well.

As the How-To Geek article points out, it's important to note that Wireshark's depth of network sniffing isn't welcome in some environments. "Don't use this tool at work unless you have permission," and carefully consider the potential consequences of sticking your sniffer in other people's business.

On the other hand, because it's an open source application, you can modify Wireshark or contribute to its development. Check out the Wireshark development portal for more details, including an extensive development guide.

Fiddler
A popular Web-debugging tool for Windows is Fiddler, the free Web-debugging proxy utility currently offered by Telerik.

Fiddler was originally a "scratch your own itch" project for Eric Lawrence. In his early days working for Microsoft on SharePoint and Office -- and learning about HTTP request traffic on the job, by necessity -- much of the Web debugging was done in the Microsoft Network Monitor (NetMon) packet sniffer. Lawrence wrote about the frustrations this caused in the Fiddler book, "Debugging with Fiddler":

"It was quickly apparent that the debugging process was overly cumbersome -- many of my colleagues were loath to use NetMon. ... Having authored a number of small utilities over the years, I was confident that I could code something to make Web debugging simpler."

As Lawrence recalls, it was "generous to call this effort primitive," but it proved to be a useful and popular tool among his peers at Microsoft. This popularity, along with interest in making the tool more robust and adding helpful features, prompted further development.

One fascinating insight that comes through Lawrence's story is this: Building the early versions of Fiddler was as much about developing his own skills and knowledge as about creating a useful tool. "The notion of writing an HTTP proxy server from scratch in .NET posed just two significant challenges: I didn't really know how HTTP worked, and I didn't know how to code in C#."

Added to the challenge of learning C# and the intricacies of the HTTP protocol, Lawrence had to figure out how to handle secure network traffic, authentication protocols, memory management and more.

Over the years, Lawrence continued working on Fiddler. "Users depended on me. As adoption grew, I didn't want to be responsible for 'abandonware.'" That sense of commitment to users who depend on the tools is one reason he keeps working on Fiddler as well as other tools he's built such as SlickRun, which Lawrence first released nearly 20 years ago.

In addition, the learning processes that kicked off development of Fiddler keep Lawrence interested in keeping it up-to-date and adding new features. "It remained interesting to me, and I was able to use it as an excuse/vehicle for learning new skills, tools, and approaches and putting them to work in Fiddler," he explains.

I was curious, though: Why did he choose to give Fiddler away? Did he ever consider retail sales?

"Originally, Fiddler was given away because all of my software is given away and Fiddler 1.0 wouldn't have been worth paying for."

Free tools are great, but should developers be able to build the tools they need (within reason)?

"I use all kinds of tools that I couldn't write," says Lawrence. "Even if I could, I wouldn't want to."

If you're interested in learning more about Fiddler, start at the Telerik support pages, which include a step-by-step video guide. There's also the book I mentioned earlier, as well as a very old MSDN Library article, " Fiddler PowerToy," both written by Lawrence himself.

I also recommend heading over to ASJava.com for its "Fiddler tutorial -- How to use Fiddler."

Charles
Another Web-debugging proxy application is Charles, created by Karl von Randow in 2000. He was working as a Web developer, which at the time entailed "quite a bit of Flash development with server-side components -- LoadVariables, XMLSocket, that sort of thing," he says.

Debugging was difficult. Mirroring the experience Lawrence had with Fiddler, von Randow solved the problems he encountered by building a tool to meet the needs of the moment. "I just knew I needed a way to see inside. I'd tried to use tcpdump, and other packet sniffers, but it was really hard. So I created Charles," he explains.

As a key part of his personal development toolbox, von Randow added new features to Charles as new requirements arose. "It started with DNS Spoofing, which was something that was always a bit of a pain when trying to test sites before the DNS changes had propagated. Then I added tools to disable browser caching, block cookies and throttle your connection. At some point I realized I could do man-in-the-middle HTTPS and bring the same capability to SSL communication."

Von Randow finally decided to release Charles publicly in 2012. "I saw how much time Charles was saving me in my day-to-day work. … I figured Charles was saving me several hours a week, so that's what convinced me that it was a tool that I should release."

I asked von Randow what he'd learned through the process of creating and updating Charles:

"One thing that I think has been really valuable is to get an insight into what people want and how they want to work when they want to be fast. Charles is a tool that you often use when you're having a bad day ... it's a helper for a problem, not the aim of what you're doing. So I try to think about what tools people need to solve problems and to solve them directly."

Another lesson learned for von Randow was to "stop fretting about how much stuff isn't perfect." You can't solve every problem or implement every feature and have it come out well. Whether you're solving client problems or maintaining your own projects, you need to choose which tasks are most important (or most interesting) and focus on doing them right.

Unlike Fiddler, Charles is a paid application (a free trial is available -- check it out). I asked von Randow to explain his thinking about charging for Charles versus offering it for free.

"Charles was always about saving time, and I was working in a service business building Web sites and time was money. I priced Charles so that it represented no more than a couple of hours of work, and I figured that people would get their money back in the first week, if not sooner."

What about my argument that developers ought to be able to build the tools they need? "Developers are so creative; there's a limitless bounty of things to create. If we all built all the tools we needed we'd never get started building anything new."

For some insight into using Charles, start with Rey Bango's "Using Web Debugging Proxies" article at Tuts+, which covers the basics of Web debugging and introduces you to both Fiddler and Charles before digging into to Charles-specific debugging techniques.

One feature that sets Charles apart from Fiddler is its ability to debug traffic for iOS apps. For details see the article, "Using Charles Proxy with Your iOS Development and HTTP Debugging," at Code with Chris and "Using Charles Proxy to Debug HTTP(S) Communication Between Server and iOS Apps" on the Noodlewerk Blog.

HttpWatch
For a slightly different perspective, take a look at what the folks at Simtec have to say about HttpWatch, its HTTP sniffer for IE, Firefox, iOS: " Seven Reasons Why Our iOS App Costs $99.99."

"That's not a typo. The paid version of the HttpWatch app really costs $ 99.99," is exactly how they start out the post, before going on to explain how they came up with the price and the development costs of keeping a team focused on ongoing support and development.

To be fair, there are basic, free versions of the app for both Windows and iOS. On the other hand, the pro version for Windows, with support and maintenance, starts at $395. Why so much?

"Unfortunately, the market for paid apps that are aimed at a technical audience like HttpWatch is very small compared to gaming apps like Angry Birds or apps with general appeal like Paper," they note in the post. "It's unreasonable to expect narrow market apps to receive the development attention they deserve if they sell for just a few dollars. The app store is littered with great apps that have never been updated for the iPhone 5 screen size or iOS 7 because they don't generate enough revenue to make it worthwhile."

I can't say whether the app is worth the price. But it is nice to have options, and in some work environments the support and maintenance contracts can be valuable for both buy-in and practical use.

Build Your Own
This column started out from the question: are developers uniquely positioned to build the tools they need? The answer was quoted from Eric Lawrence earlier in this column: "I use all kinds of tools that I couldn't write."

But what if you did want to try your hand in writing an HTTP debugger? Here are a few places to start:

Coder Bag has a very brief article, "Building a Network Sniffer in .NET," which includes a complete C# project.

Over at The Code Project, Hitesh Sharma wrote a well-received tutorial on building a sniffer, "A Network Sniffer in C#," which walks you through the code in a bit more detail.

And if you're interested in coding a bit closer to the metal, check out the article, "Create Your Own Packet Sniffer in C," at Simplest Codings.

I expect to see a wave of new competitors in this space very soon. Or not, if you have more important things to do than yak shaving. In which case you might consider trying the products of some -- or all -- of the intrepid developers highlighted here.