DevDisasters

Lost in Translation

After Wayne's employer decided to bring some outsourced code back in house, he discovered that developers in Kerbleckistan were doing some serious innovating in the science of passing parameters.

• By Mark Bowytz
• 07/23/2014

On his way back from grabbing a cup of Colombian Fair Trade from the company's caffeine/employee productivity dispenser, Wayne snatched up his printout and gave it a quick glance over, just in case he'd picked up someone else's by mistake.

Satisfied, Wayne returned to his desk, ready to dive in. Ordinarily upon receipt of a new trouble ticket, Wayne would first triage the problem by suggesting a few tried-and-true tricks that could be easily applied on the client or server side; ask smart questions about configuration (RAM, disk, CPU and so on); or alter an XML file or two to correct some settings snafu. Failing those fixes that didn't involve code, he'd bottle up the problem with a change-specification document for the offshore team in "Kerbleckistan" to handle, categorizing the root cause as likely being a software bug.

That is, until recently.

The problem with sending application troubles off to Kerbleckistan was that it didn't usually solve the reported problem; and even when it did, the cure was often worse than the disease. This was because interpersonal communication wasn't the offshore devs' strong suit, to put it mildly. Attempts to communicate with the Kerbleckistanian team was an exercise akin to trying to parse the meaning behind a translation chain going from English to German to Japanese to French and then back to English.

These communication problems frequently led to delays and, more often than not, re-submission of the same change request after it had been reported as resolved by the offshore team. Ultimately, the delays and added churn meant that more money was being thrown at small changes than should have been necessary.

So it wasn't entirely unexpected when management announced in a company-wide e-mail blast that Initrode would cease all activity with offshore, effective immediately, citing they no longer provided the "synergies" the company hoped to achieve. Considering how much revenue the application was supposed to be bringing in, positive synergy was important. In the same announcement, there was also a mention that, as part of this change, all source code would be pulled back into the organization and maintained by developers in-house -- developers who, like Wayne, usually only worked as a front-end to the offshore team.

Groans went up among the team following the announcement, but not from Wayne. Having years of experience as a seasoned application developer under his belt, he was eager to get into the application's source and figure out where things went wrong.

Not only did Wayne know that he could easily do the job of an offshore dev, he was confident that he could do it better!

A Simple Fix?
Fresh into his code analysis, something caught Wayne's eye that hinted at what the issue was when validating county information. He noticed the following method of the ASP.NET application that, despite being ASP.NET MVC 3.0, didn't use strongly typed views; instead, it passed data to the View using TempData, and retrieved data by reading the FormCollection directly:

if (this.objInitrodeAppl.UpdateSpecificCounty(countyId.ToString() + ',' + collection["ddlCurrentStatus"] + ',' + countyIsActive.ToString() + ',' + this.Session["UserId"], collection["CountyName"], collection["CountyCode"], collection["CountyState"]))
{
  this.TempData["Message"] = "Record Updated Successfully";
}

"I guess the developer felt he didn't want to have more than four parameters to the method, so he came up with the idea of passing in a comma-separated list and splitting it. That's definitely one way of doing it," Wayne said to himself.

Looking further, he found the contents of that method in, for some reason, a Windows Communication Foundation (WCF) service (although it was hosted on the same server and in the same application):

public bool UpdateSpecificCounty(string countyIDcountyStatusiSActiveuserID, string countyName, string countyCode, string CountyState)
{
  string[] getCountyinfo = countyIDcountyStatusiSActiveuserID.Split(',');
  int countyId = Convert.ToInt32(getCountyinfo[0]);
  bool status = false;

  try
  {
    this.objContainer = new InitrodeDataContainer();
    var getGetAllCounty = 
      this.objContainer.Counties.Where(c => c.CountyId == countyId).FirstOrDefault();

    if (getGetAllCounty != null)
    {
      getGetAllCounty.CountyName = countyName;
      getGetAllCounty.CountyCode = countyCode.ToUpper();
      getGetAllCounty.CountyState = CountyState;
      getGetAllCounty.CurrentStatus = getCountyinfo[1];
      getGetAllCounty.CountyOperationUserId = Convert.ToInt32(getCountyinfo[3]);
      getGetAllCounty.Countyisactive = Convert.ToBoolean(getCountyinfo[2]);
      this.objContainer.SaveChanges();
      status = true;
    }
  }
  catch (Exception exception)
  {
    FaultException<Myexception> objFaultException = this.CreatefaultException(exception);
    throw objFaultException;
  }

  return status;
}

After spending five minutes of effort (which was about how long it would take for his coffee to cool to a drinkable state), Wayne managed to add a comma check to the existing data validation logic, change the delimited string of values into actual parameters, and adjusted UpdateSpecificCounty in kind. A few minutes after that, he checked in the update, staged it, confirmed it as fixed, and deployed it into production.

Fully satisfied, Wayne enjoyed his cup of java in its entirety, thinking how this whole business of doing the job of an offshore team is a piece of cake. Just then, his e-mail inbox stuttered with multiple audible notifications of new messages. New fix tickets were arriving by the boatload, some earmarked with the dreaded "URGENT" status. What was going on here?

Make no mistake: The office was going to need a bigger brew pot.

The House of Cards Collapses
The tickets kept on rolling in the weeks following. No sooner had one bug been squashed when several more popped up in its place. Were the offshore developers superior coders after all? Hardly -- in fact, the opposite was true. The application the Kerbleckistan team had turned over functioned, but only just barely.

Held together by the digital equivalent of Popsicle sticks and duct tape, the system was sensitive to even the slightest of changes. This meant that when developers started to fix issues, they were inadvertently opening a Pandora's Box of troubles. The team concluded that the only way it was kept afloat so long was that it was being held aloft by what must have been an army of offshorers, tweaking things before they could be noticed.

Soon, the cost to keep everything going was too much to bear and, in the end, the project was canceled. Despite everyone's best efforts, the app still wouldn't work right. And even worse, there were whispers that the company had burned through almost a quarter of a million dollars on it.

Someone suggested a full-on rewrite to get things back on track, but that suggestion was quickly dismissed. Under normal circumstances, the plan might have made sense, but the cost to do so was hard to justify for a product that only managed to gain three clients.