Sunday, August 26, 2007

Summary

Project Description

FieldStat is a tool for scanning a collection of application binaries in order to understand how a particular set of classes and methods are being used in practice.

One main purpose of the tool is to support prioritizing unit test coverage based on API usage data observed in practice. Developers can view how often and how many applications use certain types and method calls in relation to unit test coverage and difficulty of testing.
This shares similar goals with clean room software engineering in that a reliability model is based on statistical likelihood of occurring. That is, place more testing effort in more likely occurring scenarios.

The tool relies on the Mono.Cecil assembly to process CIL byte-code instructions.

Delivered

Planned
  • Get up to speed with Cecil.
  • Extract call graph.
  • Identify and isolate mono framework calls.
  • Implement CodeRank algorithm.
  • Integrate with MonoCov.
  • Test and Refine application.
  • Documentation: User manual, design docs, touch-up comments
Extra
  • Command Line and Graphics Interface
  • Export Data
  • Include other statistics such as application usage count, method complexity.
  • Plugin Architecture
  • Plugin for finding Design Fragments (Set of reoccurring system calls)
  • (External) Improve Mono.Cecil tool to have better support for reconstructing statements and message chain bad code detector.

Components

Application Repository

Understanding how API calls are used in practice requires sampling actual software. Unfortunately, access to business software is limited; however, many open source applications offer a good starting point.

Over 500 projects were downloaded from the code.google.com project. The projects were selected based on the label: Mono or CSharp. In addition, 2 projects from a company were included. The projects were manually built, or a binary distribution was acquired. Some projects had to be excluded due to immaturity(not building), misclassification, and lacking the appropriate resources to build the project.

Coverage Data

The monocov tool gathers statement coverage information from the run-time execution of a Mono application. The statement coverage can be gathered in the following manner.

In the mono distribution mcs/class/corlib directory:
> make run-test RUNTIME_FLAGS="--profile=monocov:outfile=corlib.cov,+[mscorlib]"
> monocov --export-xml=/tmp/corlib-cov corlib.cov

However, the generated xml file was intended for presentation, not importing. Instead,
a new export option is introduced.

> monocov --export-fieldstat=/tmp/corlib-cov corlib.cov

Scanners and Analysis

FieldStat uses a visitor/collector pattern for gathering statistics. A visitor class walks the assemblies, classes, and methods. To gather statistics, a collector class is registered with the visitor and is notified when a particular a node of interest is visited. For instance, a collector can be notified whenever a system call is encountered.

Some default collectors included in FieldStat.
  • AppStat - Simply counts the number of system calls used per application.
  • CodeRank - Build's an application's call tree and calculates the associated code rank of each application method.
  • TypeCount - Counts the number of times a system type and system method is statically called in an application.

Running the Tool

The simpliest way run FieldStat is to specify the directory containing the coverage information (the -export-fieldstat output from monocov).

> FieldStat --coverage-path="../../../Data/Coverage Data/coverage_output" FieldStat.exe

This would output the Results.xml files in the output/ directory.

The file can be read with a XML parser or with the DataTable.ReadFile( file ) call.

A record is as follows:

<results>
<type>System.IO.Path</type>
<method>GetFileNameWithoutExtension (string)</method>
<length>1</length>
<frequency>2</frequency>
<rankedfrequency>0.30</rankedfrequency>
<appfrequency>1</appfrequency>
<coverage>1</coverage>
</results>

Screenshots

Plugins and Future Directions

Plugin System

Plugins can be created by dropping in a file named *Plugin.dll into the Plugins/ directory.

Design Fragments

An example plugin is included in the source code as the DesignFragmentPlugin project.

A design fragment is a pattern or common usage of a framework. This plugin attempts to detect candidates for design fragments by looking at common sequences of calls. This then in turn could be used to improve the framework, or serve as documentation or snippets in how to use the framework.

For example, the following series of calls was found to occur 10 times in the application repository. It looks like the user is trying to format a date in a particular way:

System.DateTime.get_Day ()
System.Int32.ToString ()
System.DateTime.get_Month ()
System.DateTime.get_Year ()
System.Int32.ToString ()
System.String.get_Length ()
System.String.Concat (string,string)
System.String.get_Length ()
System.String.Substring (int,int)
System.String.Concat (System.String[])

The plugin is written as follows:

using FieldStat.DataCollection;
using FieldStat.CodeModel;

public class DesignFragmentPlugin : AbstractPlugin
{
public override void ComputeResults(Results results, ICollection files, Hashtable htBin)
{
Visit scan = new Visit();
scan.Collectors.Register("DesignFragment", new DesignFragmentCollector());

scan.DoScan( files, htBin);
MyCollector seqs = (MyCollector)scan.Collectors["DesignFragment"];
// Process Results ...
}
}

public class DesignFragmentCollector : AbstractCollector
{
public override void OnMethodBody(MethodBody body)
{
ArrayList seq = GetSystemCallSequences(body);
if (seq.Count > 3)
{
sequences.Add(EncodeSequenceCalls( seq ));
}
}
....
}

SystemSignature

This plugin extracts the system calls made in an application. Then the resulting system call signature is compared against the other application's signatures. The results can be clustered to find common type of applications. It may also be used to judge the implementation of a well-known type of application. For example, many irc clients have been written. How divergent is yours from the other known clients? Maybe you should have taken advantage of a different architecture that you were not aware of. (Plugin under development)

8 comments:

Anonymous said...

Hello. This post is likeable, and your blog is very interesting, congratulations :-). I will add in my blogroll =). If possible gives a last there on my blog, it is about the Wireless, I hope you enjoy. The address is http://wireless-brasil.blogspot.com. A hug.

Catherine Winslet said...

People that currently have health insurance are going to be transitioned into these new plans sometime in 2014.Just Keto So the insured will be directly affected by this because the health plans they have today are going away, and they will be mapped into a new ObamaCare plan in 2014.

Illeana Menon said...

Minimize the use of lip liners; they draw moisture from the skin. Always wear lip balm, lipstick or lip gloss to keep lips moist. Lip Liner Tips

Annie Jhons said...

If you are a "water baby" or just love spending time at the beach then you will want to ensure that your makeup is waterproof. As mentioned earlier, even sweating can destroy normal makeup causing it to run all over your face. FairyMines

Annie Jhons said...

These acts have been selected based on having a clean comedy show appropriate for corporate audiences, flawless references, professionalism, and the total entertainment value of their performance. Family of Thakurganj full movie download

23 July 2019 said...

One mistake that dieters make on their weight loss plans is to skip meals or Ultra Fast Keto Boost

Annie Jhons said...

There's no question that it is healthier to prepare meals at home. You'll have more control over what goes into your food as well as the preparation. Trendy Mumi

Illeana Menon said...

The use of technology in the classroom plays a vital role in how students learn and the long-term effect of knowledge gained. When students are using technology as an educational tool they are in an active role rather than a passive role in a typical teacher-led lesson. USA Fitness Plan