When Code Coverage Betrayed Me

I’m a fan of code protection as a method to make sure that there are protecting exams. One space that I are inclined to rely closely on Code Coverage for is to catch any exams which can be now not working appropriately on account of adjustments within the manufacturing code. That usually works out nicely, however at the moment I bought betrayed by the code protection engine.

The code that I labored on contained an if assertion with a multi-step && expression.

bool IsAllWrong(int importantValue, bool b)
{
  bool a = importantValue == GetAnswer();
  bool c = false;
  bool d = false;
 
  if (!a && !b && !c && !d)
  {
    return true;
  }
  return false;
}

Of course I had exams that made the analysis fail each due to importantValue and b. So what happend later was that GetAnswer() was up to date, with out the take a look at for when importantValue being up to date. Of course (my unhealthy) that take a look at had set b to true, inflicting the analysis to fail on b, inflicting true to be returned. So the take a look at handed, however not because of the factor I wished to check. In a fancy utility, that is certain to occur now and again. But normally, the code protection scores will reveal that there’s an execution path not coated. But not this time! The reliable code protection evaluation betrayed me!

To perceive why I used to be betrayed by code protection, we’ll have to take a look at the IL code generated. It appears to be like like Visual Studios’ code protection doesn’t examine department protection, however slightly simply IL instruction protection. In most circumstances that’s the identical, however not this time.

// Initialize a, b, c & d
IL_0000:  nop         
IL_0001:  ldarg.1     
IL_0002:  ldarg.0     
IL_0003:  name        UserQuery.GetAnswer
IL_0008:  ceq         
IL_000A:  stloc.0     // a
IL_000B:  ldc.i4.0    
IL_000C:  stloc.1     // c
IL_000D:  ldc.i4.0    
IL_000E:  stloc.2     // d
// Evaluate conditional expression
IL_000F:  ldloc.0     // a
IL_0010:  brtrue.s    IL_001E
IL_0012:  ldarg.2     // b
IL_0013:  brtrue.s    IL_001E
IL_0015:  ldloc.1     // c
IL_0016:  brtrue.s    IL_001E
IL_0018:  ldloc.2     // d
IL_0019:  ldc.i4.0    
IL_001A:  ceq         
IL_001C:  br.s        IL_001F
IL_001E:  ldc.i4.0    // Run if expression was brief circuited by a, b or c being true.
IL_001F:  stloc.3     
IL_0020:  ldloc.3     
IL_0021:  brfalse.s   IL_0029
IL_0023:  nop         
IL_0024:  ldc.i4.1    
IL_0025:  stloc.s     04 
IL_0027:  br.s        IL_002E
IL_0029:  ldc.i4.0    
IL_002A:  stloc.s     04 
IL_002C:  br.s        IL_002E
IL_002E:  ldloc.s     04 
IL_0030:  ret

We can see that the analysis of the conditional expression takes the type of loading every of a, b and c after which short-circuiting the analysis if the variable is true. The downside is that if both of a, b or c is true the department instruction factors to the identical goal: IL_001E. So to get all IL directions run it’s adequate to check for 2 circumstances:

• a=b=c=d=false
• a=true; b=c=d=false

I nonetheless had each these circumstances coated. And as a result of I used to be sloppy and handed in true for b when testing importantValue the unit take a look at nonetheless handed.

So what I’ve learnt at the moment is that:

• Don’t depend on code protection to precisely calculate department protection.
• When writing exams for a fancy if assertion, it’s dangerous to be sloppy with the parameters examined after the related one. Even although they don’t seem to be related due to expression brief circuiting when every thing works, unhealthy values could make the take a look at cross on the flawed premises.