Serializing a primitive type like a bool, int, or float, is trivial: just write the data as it is (assuming that no compression is used). Serializing a pointer is different: the object it points to must be serialized first. That way deserializing the pointer simply consists of setting its value to the memory address at which the object has been reconstructed.

We can distinguish three levels of complexity in serialization, depending on how complex the pointer (and reference) graph is:

1. The pointer graph is a forest (i.e., a set of trees). Data can simply be serialized bottom up with a depth first traversal of the trees.
2. The pointer graph is a directed acyclic graph (DAG), i.e., a graph without loop. We can still serialize the data bottom up, making sure we write and restore shared data only once.
3. The pointer graph is a general graph, i.e., it may have loops. We need to write and restore data with forward references so that loops are handled properly.

It is always an option to serialize objects using your own customized code. However serialization is much more complex than a simple pretty-print method. One would like serialization to support the following features:

1. Serialization should be able to handle any pointer graph (i.e., with loops).
2. Serializing a pointer or a reference should automatically trigger the serialization of the referred object.
3. Serializing an entire data model can require a lot of code –from simple scalar fields (bool, int, float), to containers (vector, list, hash table, etc), to intricate data structures (graph, quad-tree, sparse matrices, etc). One would like templates that carry most of the burden.
4. The save and load functions must always be in sync: if the ‘save’ function is modified, the ‘load’ function must be changed appropriately. One would like that process to be automated as much as possible.
5. One should have a way of serializing objects without changing their .hpp files –this is known as non-intrusive serialization. The reason is that in many case one does not want (or one cannot) change the source files of existing libraries.
6. Serialization needs to support versioning. As objects evolve, data members are added or removed, and it is desirable to be back compatible –meaning, one can still deserialize archives from older versions into the most recent data model.
7. Serialization should be cross-platform compatible (32 and 64 bits machines, Windows, Linux, Solaris, etc).

The boost library provides a serialization that meets all the requirements above, and more:

• It is extremely efficient, it supports versioning, and it automatically serializes STL containers.
• Serialization (the save function) and deserialization (the load function) are expressed with one single template, which reduces the size of the code, and resolves the synchronization problem.
• With a little bit of help, boost serialization is also 32 and 64 bit compatible, which means that a database serialized on a 32 bit machine can be read on a 64 bit machine and conversely.
• Also boost serialization (respectively deserialization) takes an output (respectively input) argument that is very similar to a std::ostream (respectively std::istream), meaning that it can be a file on a disk, a buffer, or a socket. You can literally serialize your data over a network.

The best way to understand how to serialize with boost is to walk through increasingly complex serialization scenarios.

Basic serialization

The code for serialization, as well as an example that saves and restores simple objects, is given below.

#pragma once
// File obj.hpp

// Forward declaration of class boost::serialization::access
namespace boost {
namespace serialization {
class access;
}
}

class Obj {
public:
  // Serialization expects the object to have a default constructor
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_;
  }
private:
  int  d1_;
  bool d2_;

  // Allow serialization to access non-public data members.
  friend class boost::serialization::access;

  template<typename Archive>
  void serialize(Archive& ar, const unsigned version) {
    ar & d1_ & d2_;  // Simply serialize the data members of Obj
  }
};

The template ‘serialize’ defines both the save and load. This is achieved because the operator ‘&’ will be defined as ‘<<’ (respectively ‘>>’) for an output (respectively input) archive. Note the friend declaration to allow the save/load template to access the private data members of the objects. Also note that serialization expects the object to have a default constructor (which can be private).

#include "obj.hpp"
#include <assert.h>
#include <fstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

int main() {
  const char* fileName = "saved.txt";

  // Create some objects
  const Obj o1(-2, false);
  const Obj o2;
  const Obj o3(21, true);
  const Obj* const p1 = &o1;

  // Save data
  {
    // Create an output archive
    std::ofstream ofs(fileName);
    boost::archive::text_oarchive ar(ofs);

    // Write data
    ar & o1 & o2 & o3 & p1;
  }

  // Restore data
  Obj restored_o1;
  Obj restored_o2;
  Obj restored_o3;
  Obj* restored_p1;
  {
    // Create and input archive
    std::ifstream ifs(fileName);
    boost::archive::text_iarchive ar(ifs);

    // Load data
    ar & restored_o1 & restored_o2 & restored_o3 & restored_p1;
  }

  // Make sure we restored the data exactly as it was saved
  assert(restored_o1 == o1);
  assert(restored_o2 == o2);
  assert(restored_o3 == o3);
  assert(restored_p1 != p1);
  assert(restored_p1 == &restored_o1);

  return 0;
}

In main.cpp, we first include the files declaring the input and output text archives, where objects will be loaded from and saved to, respectively. We create an output archive (here, a file on a disk), and write three instances of class Obj, as well as a pointer to one of the instances. We then read them back and make sure we restore the data as they were. Note how the restored pointer restored_p1 points to the restored object restored_o1.

More on pointer serialization

Whenever we call serialization on a pointer (or reference), this triggers the serialization of the object it points to (or refers to) whenever necessary. So we do not need to explicitly serialize pointed objects as boost serialization will make sure the appropriate objects reached in the pointers graph are serialized.

For instance, the code below shows that serializing the pointer p1 triggers the serialization of o1, the object it point to. When restoring the pointer restored_p1, we automatically create a clone of the object o1.

#include "obj.hpp"
#include <assert.h>
#include <fstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

int main()
{
  const char* fileName = "saved.txt";

  // Create one object o1.
  const Obj o1(-2, false);
  const Obj* const p1 = &o1;

  // Save data
  {
    // Create an output archive
    std::ofstream ofs(fileName);
    boost::archive::text_oarchive ar(ofs);
    // Save only the pointer. This will trigger serialization
    // of the object it points too, i.e., o1.
    ar & p1;
  }

  // Restore data
  Obj* restored_p1;
  {
    // Create and input archive
    std::ifstream ifs(fileName);
    boost::archive::text_iarchive ar(ifs);
    // Load
    ar & restored_p1;
  }

  // Make sure we read exactly what we saved.
  assert(restored_p1 != p1);
  assert(*restored_p1 == o1);

  return 0;
}

When deserializing a pointer, the object it points to will be automatically deserialized if this object has not been deserialized yet. This means that one should not attempt to deserialize an object after a pointer to this object has been deserialized. The reason is that once the pointer deserialization has forced the object deserialization, one cannot rebuild this object at a different address.

#include "obj.hpp"
#include <fstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

int main()
{
  const char* fileName = "saved.txt";
  std::ofstream ofs(fileName);

  // Create one object o1 and a pointer p1 to that object.
  const Obj o1(-2, false);
  const Obj* const p1 = &o1;

  // Serialize object, then pointer.
  // This works fine: after the object is deserialized, we can
  // deserialize the pointer by assigning it to the object’s address.
  {
    boost::archive::text_oarchive ar(ofs);
    ar & o1 & p1;
  }

  // Serialize pointer, then object.
  // This does not work: once p1 has been serialized, the object
  // has already been deserialized and its address cannot change.
  // This will throw an instance of 'boost::archive::archive_exception'
  // at runtime.
  {
    boost::archive::text_oarchive ar(ofs);
    ar & p1 & o1;
  }

  return 0;
}

In the example above, the second serialization will result in a runtime error:

ocoudert@MyMacBookPro $ a.out
terminate called after throwing an instance of 'boost::archive::archive_exception'
    what():  pointer conflict
Abort trap
coudert@MyMacBookPro $

This means that when pointers need to be serialized, we should never explicitly serialize the objects they point to.

Explicit save and load function definitions

We need an explicit definition of the save and load functions whenever they are not fully symmetric. This is typical when versioning is involved. Note the use of the macro BOOST_SERIALIZATION_SPLIT_MEMBER(), which is responsible for calling save/load when using an output/input archive.

#pragma once

#include <boost/serialization/split_member.hpp>

class Obj {
public:
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_;
  }

private:
  int  d1_;
  bool d2_;

  friend class boost::serialization::access;

  template<class Archive>
  void save(Archive & ar, const unsigned int version) const {
    ar & d1_ & d2_;
  }

  template<class Archive>
  void load(Archive & ar, const unsigned int version) {
    ar & d1_ & d2_;
  }

  BOOST_SERIALIZATION_SPLIT_MEMBER()
};

Serialization of C-strings

A C-string cannot be directly serialized because it assumes a specific interpretation of a char*, namely an array of char terminated by a null character (‘\0’). Thus we need to explicitly serialized C-string. The class below is a simple helper to serialize C-strings (note that this can be optimized by avoiding the construction of the sdt::string).

#pragma once
// File SerializeCStringHelper.hpp

#include <string>
#include <boost/serialization/string.hpp>
#include <boost/serialization/split_member.hpp>

class SerializeCStringHelper {
public:
  SerializeCStringHelper(char*& s) : s_(s) {}
  SerializeCStringHelper(const char*& s) : s_(const_cast<char*&>(s)) {}

private:

  friend class boost::serialization::access;

  template<class Archive>
  void save(Archive& ar, const unsigned version) const {
    bool isNull = (s_ == 0);
    ar & isNull;
    if (!isNull) {
      std::string s(s_);
      ar & s;
    }
  }

  template<class Archive>
  void load(Archive& ar, const unsigned version) {
    bool isNull;
    ar & isNull;
    if (!isNull) {
      std::string s;
      ar & s;
      s_ = strdup(s.c_str());
    } else {
      s_ = 0;
    }
  }

  BOOST_SERIALIZATION_SPLIT_MEMBER();

private:
  char*& s_;
};

A simple example of its usage is as follows.

#include "SerializeCStringHelper.hpp"
#include <assert.h>
#include <fstream>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

int main()
{
  const char* fileName = "saved.txt";
  const char* str = "This is an example a C-string";

  // Save data
  {
    // Create an output archive
    std::ofstream ofs(fileName);

    boost::archive::text_oarchive ar(ofs);
    // Save
    SerializeCStringHelper helper(str);
    ar & helper;
  }

  // Restore data
  char* restored_str;
  {
    // Create and input archive
    std::ifstream ifs(fileName);
    boost::archive::text_iarchive ar(ifs);

    // Load
    SerializeCStringHelper helper(restored_str);
    ar & helper;
  }

  // Make sure we read exactly what we saved
  assert(restored_str!= str);
  assert(strcmp(restored_str, str) == 0);

  return 0;
}

Non-intrusive serialization

So far the serialization code is added in the class definition. A non-intrusive serialization, outside of the class, might be preferable. For instance we would like to serialize a class from a library without altering the library’s hpp file. This is easy when the data members are public:

#pragma once

class Obj {
public:
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_;
  }

public:
  int  d1_;
  bool d2_;
};

namespace boost {
namespace serialization {

template<typename Archive>
void serialize(Archive& ar, Obj& o, const unsigned int version) {
  ar & o.d1_ & o.d2_;
}

} // namespace serialization
} // namespace boost

If we want to protect the data members, the code is a bit more complicated because the serialization template needs to be declared as a friend. This requires a forward declaration of the template.

#pragma once

//// Declaration of the template
class Obj;

namespace boost {
namespace serialization {

template<typename Archive>
void serialize(Archive& ar, Obj& o, const unsigned int version);

} // namespace serialization
} // namespace boost

//// Definition of the class
class Obj {
public
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_;
}

private:
  int  d1_;
  bool d2_;

  // Allow serialization to access data members.
  template<typename Archive> friend
  void boost::serialization::serialize(Archive& ar, Obj& o, const unsigned int version);
};

//// Definition of the template
namespace boost {
namespace serialization {

template<typename Archive>
void serialize(Archive& ar, Obj& o, const unsigned int version) {
ar & o.d1_ & o.d2_;
}

} // namespace serialization
} // namespace boost

Non-intrusive explicit save and load function definitions

This combines the two previous serialization styles, except that the include file and macro are different. For the sake of simplicity, we give the version for public data members.

#pragma once

#include <boost/serialization/split_free.hpp>

class Obj {
public:
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_;
  }

public:
  int  d1_;
  bool d2_;
};

namespace boost {
namespace serialization {

template<class Archive>
void save(Archive & ar, const Obj& o, const unsigned int version) {
  ar & o.d1_ & o.d2_;
}

template<class Archive>
void load(Archive & ar, Obj& o, const unsigned int version) {
  ar & o.d1_ & o.d2_;
}

} // namespace serialization
} // namespace boost

BOOST_SERIALIZATION_SPLIT_FREE(Obj)

Serialization of STL containers

The boost library comes with templates to automatically serialize STL containers, as well as some STL objects (e.g., std::string). Instead of saving/loading a vector with the following code:

template<typename Archive>
void save(Archive& ar, const std::vector<Obj>& objs, const unsigned version) {
  ar << objs.size();
  for (size_t i = 0; i < objs.size(); ++i) {
    ar << objs[i];
  }
}

template<typename Archive>
void load(Archive& ar, std::vector<Obj>& objs, const unsigned version) {
  size_t size;
  ar >> size;
  objs.resize(size);
  for (size_t i = 0; i < size; ++i) {
    ar >> objs[i];
  }
}

One simply writes:

#include <boost/serialization/vector.hpp>

template<typename Archive>
void serialize(Archive& ar, std::vector<Obj>& objs, const unsigned version) {
  ar & objs;
}

All the STL containers are supported using the appropriate include files:

#include <boost/serialization/array.hpp>
#include <boost/serialization/vector.hpp>
#include <boost/serialization/hash_map.hpp>
#include <boost/serialization/hash_set.hpp>
#include <boost/serialization/list.hpp>
#include <boost/serialization/slist.hpp>
#include <boost/serialization/map.hpp>
#include <boost/serialization/set.hpp>
#include <boost/serialization/bitset.hpp>
#include <boost/serialization/string.hpp>

Serialization of base class

When a class inherits from another, the base class needs to be serialized as well.

#include <boost/serialization/base_object.hpp>

class Base {
public:
  Base() : c_('\0') {}
  Base(char c) : c_(c) {}
  bool operator==(const Base& o) const { return c_ == o.c_; }

private:
  char c_;

  friend class boost::serialization::access;

  template <typename Archive>
  void serialize(Archive& ar, const unsigned version) {
    ar & c_;
  }
};

class Obj : public Base {
private:
  typedef Base _Super;
public:
  Obj() : _Super(), d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : _Super('a'), d1_(d1), d2_(d2) {}
  bool operator==(const Obj& o) const {
    return _Super::operator==(o) && d1_ == o.d1_ && d2_ == o.d2_;
  }

private:
  int  d1_;
  bool d2_;

  friend class boost::serialization::access;

  template <typename Archive>
  void serialize(Archive& ar, const unsigned version) {
    ar & boost::serialization::base_object<_Super>(*this);
    ar & d1_ & d2_;
  }
};

Versioning

We want maintain back-compatibility when the class Obj evolves. For instance, if a new data member ‘ID_’ is added, we want to read an old archive and build new Obj, with the missing data member taking the default value.

#pragma once

#include <boost/serialization/split_member.hpp>
#include <boost/serialization/version.hpp>

class Obj {
public:
  Obj() : d1_(-1), d2_(false), ID_(0) {}
  Obj(int d1, bool d2, unsigned ID id) : d1_(d1), d2_(d2), ID_(id) {}
  bool operator==(const Obj& o) const {
    return d1_ == o.d1_ && d2_ == o.d2_ && ID_ == o.ID_;
  }

private:
  int  d1_;
  bool d2_;
  unsigned ID_;

  friend class boost::serialization::access;

  template<class Archive>
  void save(Archive & ar, const unsigned int version) const {
    ar & d1_ & d2_ & ID_;
  }

  template<class Archive>
  void load(Archive & ar, const unsigned int version) {
    ar & d1_ & d2_;
    // If archive’s version is 0 (i.e., is old), ID_ keeps
    // its default value from the new data model,
    // else we read ID_’s value from the archive.
    if (version > 0) {
      ar & ID_;
    }
  }

  BOOST_SERIALIZATION_SPLIT_MEMBER()

};

Serialization of const data or objects

Attempting to serialize a const data or object triggers a long trail of error messages, which includes something that looks like:

[snip]

/opt/local/include/boost/archive/detail/check.hpp:162: error:
  invalid application of ‘sizeof’ to incomplete type ‘boost::STATIC_ASSERTION_FAILURE<false>‘

[snip]

/opt/local/include/boost/archive/basic_text_iprimitive.hpp:88: error:
  ambiguous overload for ‘operator>>‘ in
  ‘((boost::archive::basic_text_iprimitive<std::basic_istream<char,
       std::char_traits<char> > >*)this)->boost::archive::basic_text_iprimitive<std::basic_istream<char,
         std::char_traits<char> > >::is >> t’

This means that the input archive expects the recipient of the data to be non-const. Thus const data members must be const_cast<>()’ed to be serialized. For example:

#pragma once

#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

class Obj {
public:
  Obj() : d1_(-1), d2_(false) {}
  Obj(int d1, bool d2) : d1_(d1), d2_(d2) {}

private:
  const int d1_;
  bool d2_;

  // Allow serialization to access data members.
  friend class boost::serialization::access;

  template<typename A>
  void serialize(A& ar, const unsigned version) {
    ar & const_cast<int&>(d1_) & d2_;
  }
};

Text, XML, and binary archives

The text archive is an ASCII file that is somewhat human readable. There are other archive types available in boost/archive/*.hpp, e.g.:

// Text archive that defines boost::archive::text_oarchive
// and boost::archive::text_iarchive
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/text_oarchive.hpp>

// XML archive that defines boost::archive::xml_oarchive
// and boost::archive::xml_iarchive
#include <boost/archive/xml_oarchive.hpp>
#include <boost/archive/xml_iarchive.hpp>

// XML archive which uses wide characters (use for UTF-8 output ),
// defines boost::archive::xml_woarchive
// and boost::archive::xml_wiarchive
#include <boost/archive/xml_woarchive.hpp>
#include <boost/archive/xml_wiarchive.hpp>

// Binary archive that defines boost::archive::binary_oarchive
// and boost::archive::binary_iarchive
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>

The text and XML archives are portable across 32 and 64 bits platforms.

Having a binary archive that is portable between 32 and 64 bits is not trivial, because C++ does not specify exactly the size of primitive types. For instance a long is usually 4 bytes on a 32 bits machine, and 8 bytes on a 64 bits machine. In practice though it is pretty portable –there is a non-official version for a portable binary archive.

Related posts:

1. How to write abstract iterators in C++
2. How to make software deterministic

Refining this definition, we should consider whether a program produces the same result on any platform (32 and 64 bits machines, running Windows, Mac OS, Linux, Solaris, etc). Or whether the program is insensitive to the form of its inputs. For example, the problem of generating the shortest route to visit all the capitals of Europe should not depend on how the map of Europe is entered, nor it should depend on which language is used to name the capitals.

Determinism is obviously very desirable. For the user, a non-deterministic program can be confusing and frustrating. For the developer, a non-deterministic program is extremely hard to test and debug, since bugs and specific configuration cannot be easily reproduced.

Repeatability looks like a given for most applications. For instance, if we add two numbers in a spreadsheet, we expect the same result no matter how many times we perform this operation and regardless of the platform we run on (PC, Mac, etc). Or if we run a spell checker several times, we expect it to flag the very same errors.

But it is not that obvious for more complex applications. This is especially true when there are multiple solutions to a problem, or when heuristics are used to produce a result –because an exact solution is too computationally expensive. For example, it is not uncommon to see slightly different outcomes when running the same EDA synthesis or P&R tool on the same input several times.

Even more, a user would like to see the same result when only minor changes are applied to the input. For instances, running a P&R tool on two netlists that differ only by the names of their cells should produce exactly the same result –a P&R tools should produce a result that only depend on the netlist structure. But experience shows that industrial synthesis and P&R tool does not meet that requirement. Closest to software, it is not uncommon to generate slightly different object codes with gcc by changing the names of a few variables.

Among the causes of non-deterministic response, we can distinguish the following types:

1. A random number generator
2. Reading an uninitialized data
3. A race condition on concurrent threads
4. An unordered iteration that is assumed ordered
5. A total order that depends on memory address
6. A total order that depends on time stamps
7. A total order that depends on a non-canonical labeling

1. Random number generator

There are a lot of applications that use stochastic processes (e.g., simulated annealing, genetic algorithms, Monte-Carlo simulations), but that we would like to be repeatable. Using a pseudorandom number generator with a known seed makes possible to reproduce the same long sequence of seemingly random numbers over and over again.

Note that some applications (e.g., gaming, cryptography, statistical sampling) require a non-deterministic behavior. In that case the seed of the random number generator must be an always-changing value, for example the host’s current time.

There are more deliberate efforts to produce true random values by relying on natural, chaotic events. For instance Lavarand produces random numbers by hashing the frames of a video stream of lava lamps. HotBits generates random bits by timing successive pairs of radioactive decays detected by a Geiger-Müller tube interfaced to a computer. Random.org uses variations in the amplitude of atmospheric noise recorded with a normal radio.

2. Uninitialized or random data read

Initialized data may not exist in languages that have systematic default values and no memory management control, as opposed to high performance languages like C/C++.

Finding and fixing this kind of issues is relatively simple. For instance, tools like Purify and Valgrind can report when a C/C++ code reads arbitrary values in memory. To use Purify’s terminology, such errors are UMR (Uninitialized Memory Read), ABR (Array Bound Read, i.e., dereferencing an array outside of its bounds), and FMR (Free Memory Read). These defects all consist in reading some random value in memory.  The code below illustrates some of these errors.

#include
#include 

int main() {
  bool large;
  int size = 100;

  if (large) {                    // UMR
    size *= 10;
  }

  char* const str    = new char(size);
  char* const middle = str + size/2;
  // Set the end-of-string.
  str[size - 1] = '\0';
  // The loop intends to fill up the string str with 'a'.
  // But this loop is faulty, because *++c is used instead of *c++.
  char* c = str;
  for (int i = 1; i < size; ++i, *++c = 'a');

  printf("%c\n", str[0]);         // UMR
  printf("%c\n", str[1]);         // Ok, will print 'a'.
  printf("%c\n", str[2]);         // Ok, will print 'a'.
  printf("%c\n", str[size - 1]);  // Ok, but will print 'a'
                                  // instead of the expected '\0'.
  printf("%c\n", str[size]);      // ABR
  printf("%lu\n", strlen(str));   // UMR, because we overwrote
                                  // the final '\0' in the loop.
  delete [] str;
  printf("%c\n", *middle);        // FMR

  return 0;
}

Note that an ABW (Array Bound Write, i.e., writing outside of an array’s bounds), FMW (Free Memory Write), FNH (Freeing Non-Heap memory) and FUM (Freeing Unallocated Memory), although severe bugs also reported by dynamic analysis tools, are not an original source of non-determinism: they consistently reproduce the same bug.

3. Thread races

Thread races are difficult to detect, and fixing them can be very costly. A typical example is when one thread writes a value at some address, and another thread reads the value at that address. Depending on which thread access the address first, the outcome of the program will be different. Two threads performing a non-atomic write at the same address simultaneously results in some unpredictable value.

One can use a mutex to prevent conflicting read/write for non-atomic operations. But racing threads (e.g., who reads/writes first) must be resolved with synchronization, which can be quite complicated. Moreover it can hurt performances.

4. Iteration on unordered data

Iterating data with some random order can make a program non repeatable. This pattern is often encountered, and is easy to fix.

For example an algorithm produces a result via a visitor that assumes a total order. The developer uses an incorrect visitor, which enumerates data in a random order, usually depending on the memory allocation of the data container. E.g., instead of using a std::set as a container, the developer uses a std::hash_set (or a tr1::unordered_set instead of a tr1::ordered_set). Forcing a total order on the data fixes the problem.

Note that the fix may be incomplete if it simply transforms a type (4) non-determinism into a type (5) or (6) non-determinism, which we discuss below.

5. Ordering by pointer value

This type of non-determinism is extremely common. For instance, a developer uses a tr1::ordered_set as a container of pointers, and feels that the visitor is deterministic. It is indeed deterministic, but only w.r.t. the memory addresses allocated to the data, which depend on factors out of the application’s control.

One way of addressing the problem is to force a specific memory addressing scheme, but that requires a very fine control of the memory allocator and is therefore complicated.  A more common way consists in assigning a unique ID to an object at the time of its creation. The ID can be a 32 bit unsigned integer that is incremented for every new object. A total ordering, independent from the objects’ actual memory addresses, is then obtained from the IDs. It is a simple solution, as long as one can afford the extra 4 bytes for every object. ID-based sorting with no memory penalty can be obtained using custom memory allocators.

6. Ordering by time stamps

Note though that the total ID-based ordering described above is exactly the order of creation of the objects. It is no different from an order that depends on time stamps. Therefore two equal sets of inputs that only differ in the order will be visited in a different order, which can lead to different results. This leads us to the type (7) of non-determinism.

7. Ordering induced by a non-canonical labeling

Type (7) non-determinism often goes unrecognized, or is simply ignored. The idea is that as long as the same input is given to a program (but possibly in a different order or form), the output should be the same. If the input can somehow be normalized to a form that captures the notion of “same input”, then the program can be made insensitive to the format of the input. That is of course assuming that the normalization process run time penalty is not too high.

This normalization process is better defined as canonization. Formally, let O be a set of objects, and let EQ be an equivalence relation that captures the notion of “same” on these objects. A function Canon maps an object onto its canonical form, and is such that for any two objects o1 and o2, o1 and o2 are the same (i.e., o1 EQ o2) if and only if Canon(o1) = Canon(o2).

For instance, a set of integers can be represented by a number of containers (a list, an array, a hash set, a binary tree, etc). A canonical form can simply consist in sorting the integers. Two sets that are equal because they contain the very same integers, but that are initially given in different orders and forms, will end up in the same canonical form. Since sorting is an O(n log n) algorithm, this is an efficient canonization.

Canonization can be much more costly. A Boolean function can be represented in many ways, e.g., with a truth table, a Conjunctive Normal Form (CNF), a Disjunctive normal form (DNF), a decision diagram, etc (see below). Boolean function canonization is at least NP-hard, since it solves the satisfiability problem (SAT). In practice this means that Boolean function canonization algorithms have an exponential complexity.

Canonization can also be elusive. Let us consider the problem of drawing a graph in some aesthetic way (e.g., such that the nodes are evenly distributed and such that there is a minimum number of crossing edges). One would like the graph to be drawn the very same way, regardless of its representation (adjacency list or adjacency matrix), and regardless of the order the adjacency information is given. For instances the three graphs below, although looking different, are exactly the same, and can be drawn without any edge crossing as shown on the right side.

Graph canonization is also known as graph labeling. It is at least as hard as graph isomorphism, one of these rare problems that are in NP but that are not known to be NP-complete or in P. Although all existing graph canonization algorithm have an exponential worst-case complexity, it is believed that graph canonization can be done in polynomial time.

Conclusion

The most common  cause for non-determinism is related to some unreliable data order. The ultimate solution to make a program insensitive to the form of its input is to canonize its input as a pre-processing step. This proves to be a challenging and costly task in some cases. Whenever possible, canonization (or some imperfect normalization) goes a long way to make the application consistently repeatable.

Related posts:

1. What is software quality?
2. Test-driven design, a methodology for low-defect software
3. How to write abstract iterators in C++

Some quality criteria are objective, and can be measured accordingly. Some quality criteria are subjective, and are therefore captured with more arbitrary measurements.

The table below lists the most obvious software quality criteria, as well as some lesser known.

By definition the internal quality (code characteristics) is a concern to the developer only, while all the external quality aspects (coming from using the software) are critical to the end user. However the developer has also interests in performances (speed, space, network usage) and determinism, because they make testing the software easier. Developers treat ease-of-use, back-compatibility, security, and power consumption as requirements.

It is important to consider how difficult it is to measure each of these criteria. It can be difficult because there is no simple variable to look at, or because the measurement process is costly, or because it requires a complex infrastructure. For instance, speed has an objective measurement that is easy to measure. Power consumption has a simple measurement (how many µW the application consumes), but it is complex to measure. Security is difficult and costly to estimate.

Features. This is the very reason for the software to be written: to provide a service. By feature we really mean the output produced by the software –e.g., a numerical result, a string, a screen shot, a web page, an audio, etc–, regardless of the performances (speed, memory).

Speed. How quickly does the application provide the service? The user experiences the actual time elapsed between the moment she request the service, and the moment the service is delivered. The real elapsed time, or wall time, is the sum of the CPU time, system time, and network latency. Thus the developer should not only focus on the CPU time (how much time the CPU actually spends on executing the program). The CPU time can easily be overshadowed by disk access (a write on the disk is very costly), swapping (due to an excessive virtual memory size), or time spent by the network (latency issue, or too many round trips).

Space. How much RAM and disk space is taken by the application? The aggregate numbers are important –peak memory, virtual memory size, etc. But even more so, how often do we move data that triggers a CPU cache miss or a disk write, has a dominant impact on the speed of the application. A mediocre data design can lead to very poor performances.

Network usage. It is a matter of bandwidth and latency. Mismanaging sockets and channels can lead to unnecessary extra time spent in opening and closing sockets, handshakes, and round trips. As for memory, caching techniques can be used to reduce consuming network resources.

Stability. How often does one need to patch the software to fix problems? For the user, this is an inconvenience. For the developer, it means that the code is fragile and might benefit from better testing or partial rewrite.

Robustness. How often does the application stale, freeze, or crash? How tolerant is it to extreme conditions –limited CPU and memory/disk/network resources, corner cases, system failure or unresponsive 3^rd-party resources? This aspect is strongly related to testability and coverage.

Ease-of-use. It can be a very subjective factor, hard to quantify. It includes user documentation, clarity of the error message, management of exceptions, and recovery after failure.

Determinism. Also known as repeatability: does the program produce the very same result given the same input? There are many reasons for which a program can exhibit a non-repeatable behavior. A non-repeatable behavior is confusing and frustrating for the user. This also makes the program very difficult to test and debug. Repeatability is strongly dependent on a good data model design.

Back-compatibility. Can a new version of the application be used with an older version’s data? It is essential to the user, because a new version should not require a costly migration of the existing data.

Security. Who is authorized to access the data? Can the data processed by the application be compromised? This is a crucial aspect of many applications, and it is getting more and more difficult to assess with the dissemination of mobile and web-based software.

Power consumption. It is increasingly important with mobile applications, as a program may have to consider how it manages the device’s power producers and consumers (battery, cores, wireless, screen, audio), and not to rely entirely on the operating system.

Test coverage. What is the proportion of code that is executed by some unit or regression test? This is measured by the number of lines, number of functions, and number of control branches that are exercised by the tests. Usually one expects coverage of at least 85% for any moderately complex application. In practice reaching high coverage can be achieved only if testability is high, which has deep implication on the architecture and development methodology.

Testability. An often overlooked or simply ignored aspect of code development, testability is the ability to trigger any specific line of code or branching condition. Highly testable code requires a discipline of architecture and development that is difficult to find. It very costly to fix poorly testable software, as this requires major redesign. This justifies major investment in software architecture, design, and development methodologies.

Portability. Can the application run on 32 and 64 bits machines? Should it run on a mobile phone? Does it run on multiple OS (e.g., Windows, Linux, Mac OS-X, Solaris, iOS, Android, RIM)? Does it run smoothly on all web browsers (IE, Firefox, Chrome, Safari, Opera)?

Thread-safeness. Is a specific component thread-safe? Can two threads collide on non-atomic operations? Can the application get into a deadlock? As concurrency is still mostly the result of a manual process (there no compiler that automatically parallelizes the code), these questions are critical to ensure the good functioning of a program, as well as its performance –it is not rare to see the a program running slower when two many threads are available, as the cost of synchronization can become dominant.

Conciseness. Also known as compactness. Is there any dead code, or duplicated code? Is the code shared and factorized properly? A compact code usually means faster compilation and smaller binary size. Also compactness naturally leads to fewer bugs, because the number of bugs is historically constant w.r.t. code size.

Maintainability. How easy it is to debug the code? How fast is it to provide a fix? How quickly can a new developer understand the code? Maintainability is a very important aspect, quite difficult to quantify. Maintainability is increased with good testability and flexible (abstract) design.

Documentation. This is a pretty subjective topic. Some people claim that a separate documentation written in plain English is necessary. Some others state that at least 30% of the code should be comments. Some finally argue that the code itself is the best documentation –the names of the types, classes, functions and arguments, together with plenty of assertions.

Legibility. Also known as readability. This is another subjective topic. It is about how easy it is to read the code. Guidelines are established to unify the style of the code, so that a developer can easily read code written by another developer. Code guidelines abound, and they go from a small set of directives, to a full set of rules that specify every syntactical aspect of the language. For example, see Industrial Strength C++, High Integrity C++, Google C++ Style Guide, and many more.

Scalability. How easy it is to extend a feature? Or to add a new one? Or to add extra cores, or increase the size of the cluster the application runs on? Again, this is all about software architecture and anticipating future needs.

Software quality is the result of the user experience. But software quality should not and cannot be a reactive action to external defects. Software quality is built from the ground up, with design and development methodologies, and with a special focus on testability, coverage, and flexibility.

Related posts:

1. How to make software deterministic
2. Test-driven design, a methodology for low-defect software
3. How to write abstract iterators in C++

When developing in C++, an impeccable API is a must have: it has to be as simple as possible, abstract, generic, and extensible. One important generic concept that STL made C++ developers familiar with is the concept of iterator.

An iterator is used to visit the elements of a container without exposing how the container is implemented (e.g., a vector, a list, a red-black tree, a hash set, a queue, etc). Iterators are central to generic programming because they are an interface between containers and applications. Applications need access to the elements of containers, but they usually do not need to know how elements are stored in containers. Iterators make possible to write generic algorithms that operate on different kinds of containers.

For example, the following code snippet exposes the nature of the container –a vector.

     void process(const std::vector<E>& v)
     {
         for (unsigned i = 0; i < v.size(); ++i) {
             process(v[i]);
         }
     }

If we want to have the same function operating on a list, we have to write a separate function. Or if we later decide that a list or a hash set is more appropriate as a container, we need to rewrite the code everywhere we access the vector. This may require a lot of changes in many files. Contrast this container-specific visitation scheme to the following:

     template <typename Container>
     void process(const Container& c)
     {
         typename Container::const_iterator itr = c.begin();
         typename Container::const_iterator end = c.end();
         for (; itr != end; ++itr) {
             process(*itr);
         }
     }

Using the notion of iterator, we have a generic processing of a container ‘c’, whether it is a vector, a list, a hash set, or any data structure that provides iterators in its API. Even better, we can write a generic process function that only takes an iterator range, without assuming that the container has a begin() and end() method:

     template <typename Iterator>
     void process(Iterator begin, Iterator end)
     {
         for (; itr != end; ++itr) {
             process(*itr);
         }
     }

An STL iterator is a commodity that behaves as a scalar type:

• It can be allocated on the heap
• It can be copied
• It can be passed by value
• It can be assigned to

The essence of an iterator is captured by the following API.

     template <typename T>
     class Itr {
     public:
         Itr();
         ~Itr();
         Itr(const Itr& o);                   // Copy constructor
         Itr& operator=(const Itr& o);        // Assignment operator
         Itr& operator++();                   // Next element
         T&   operator*();                    // Dereference
         bool operator==(const Itr& o) const; // Comparison
         bool operator!=(const Itr& o) const { return !(*this == o); }
     }

Usually the container will provide a begin() and end() method, which build the iterators that denote the container’s range. Writing these begin/end methods is an easy task if the container is derived from a STL container, if the container has a data member that is an STL container, or if the iterator is a scalar type, like a pointer or an index.

It is more complicated if we want iterators that dereference to the same type of object, but that must visit several containers, possibly of different types, or iterators that visit containers in different manners. For instance let us assume that we have objects with some property (say, a color) stored in several containers, some of them of different types. We would like to visit all the objects, independently of the number of containers and their type, or we would like to visit objects of a given color, or we would like to visit objects that satisfy some predicate:

     class E;

     Itr<E> begin(); // This give the range to visit
     Itr<E> end();   // all the elements of type E      

     Itr<E> begin(const Color& color); // Same as above but only for the
     Itr<E> end(const Coir& color);    // elements of the given color      

     class Predicate {
     public:
         bool operator()(const E& e);
     };      

     Itr<E> begin(Predicate& p); // Same as above but only for the
     Itr<E> end(Predicate& p);   // elements that satisfy the predicate

In this case the iterator is more complex than a scalar type like a pointer or an index: it needs to keep track of which container it is currently visiting, or which color or predicate it needs to check. In general, the iterator may have data members so that it can fulfill its task. Also we want to factorize the code and reuse general purpose iterators’ methods when writing more targeted iterators –e.g., visiting elements of a specific color should make use of the next-element method Itr<E>::operator++(). This can be done by having Itr<E> be a virtual class, and having derived classes to implement the different iterators. For example:

     class E {
     public:
         Color& color() const;
     };      

     template <typename E>
     class ColoredItr<E> : public Itr<E> {
     private:
         typedef Itr<E> _Super;
     public:
         ColoredItr<E>(const Color& color) : Itr<E>(), color_(color) {}
         virtual ~ColoredItr<E>;
         virtual ColoredItr<E>& Operator++() {
            for (; _Super::operator*().color() != color_; _Super::operator++());
            return *this;
         }
     private:
         Color color_;
    };

We would like a generic iterator that meets all the requirements described above:

• It can be allocated on the heap
• It can be copied
• It can be passed by value
• It can be assigned to
• It dereferences to the same type
• It can visit several containers
• It can visit containers of different types
• It can visit containers in arbitrary manners

This can be implemented as follows.

     template<typename E>
     class ItrBase {
     public:
         ItrBase() {}
         virtual ~ItrBase() {}
         virtual void  operator++() {}
         virtual E&    operator*() const { return E(); }
         virtual ItrBase* clone() const { return new ItrBase(*this); }
         // The == operator is non-virtual. It checks that the
         // derived objects have compatible types, then calls the
         // virtual comparison function equal.
         bool operator==(const ItrBase& o) const {
             return typeid(*this) == typeid(o) && equal(o);
         }
     protected:
         virtual bool equal(const ItrBase& o) const { return true; }
     };      

     template<typename E>
     class Itr {
     public:
         Itr() : itr_(0) {}
         ~Itr() { delete itr_; }
         Itr(const Itr& o) : itr_(o.itr_->clone()) {}
         Itr& operator=(const Itr& o) {
             if (itr_ != o.itr_) { delete itr_; itr_ = o.itr_->clone(); }
             return *this;
         }
         Itr&  operator++() { ++(*itr_); return *this; }
         E&    operator*() const { return *(*itr_); }
         bool  operator==(const Itr& o) const {
             return (itr_ == o.itr_) || (*itr_ == *o.itr_);
         }
         bool  operator!=(const Itr& o) const { return !(*this == o); }      

     protected:
         ItrBase<E>* itr_;
     };

The ItrBase class is the top class of the hierarchy. Itr is simply a wrapper on a pointer to an ItrBase, so that it can be allocated on the heap –the actual implementation of the class deriving from ItrBase can have an arbitrary size. Note how the Itr copy and assignment operators are implemented via the ItrBase::clone() method, so that Itr behaves as a scalar type. Last but not least, the (non-virtual) ItrBase::operator== equality operator first checks for type equality before calling the (virtual) equality method equal on the virtual subclass. The reason ItrBase is not a pure virtual is that it can conveniently be used to denote an empty range, i.e., the range (ItrBase(), ItrBase()) is empty.

Iterators on containers of elements of type E just need to derive from ItrBase<E>, and a factory providing the begin() and end() methods for any specialized iterator returns object of type Itr<E>.

For example, let us assume that we have a container c of E’s, and that we want an iterator to visit (1) all the elements of c, possibly with repetition; (2) all the elements of c without repetition. This can be done as follows.

    class E;

    class ItrAll : public ItrBase<E> {
    private:
        typedef ItrAll     _Self;
        typedef ItrBase<E> _Super;
    public:
        ItrAll(Container& c) : _Super(), c_(c) {}
        virtual ~ItrAll() {}
        virtual void  operator++() { ++itr_; }
        virtual E&    operator*() const { return *itr_; }
        virtual ItrBase<E>* clone() const { return new _Self(*this); }
    protected:
        virtual bool equal(const ItrBase<E>& o) const {
            // Casting is safe since types have been checked by _Super::operator==
            const _Self& o2 = static_cast<const _Self&>(o);
            return &c_ == &o2.c_ && itr_ == o2.itr_;
        }
    protected:
        Container&          c_;
        Container::iterator itr_;
    };     

    class ItrNoRepeat : public ItrAll {
    private:
        typedef ItrNoRepeat _Self;
        typedef ItrAll      _Super;
    public:
        ItrNoRepeat (Container& c) : _Super(c) {}
        virtual ~ItrNoRepeat () {}
        virtual void  operator++() {
            _Super::operator++(); // Go to the next element then
            // look for an element that has not been visited yet.
            for (; itr_ != c_.end(); _Super::operator++()) {
                E& e = _Super::operator*();
                if (visited_.find(e) == visited_.end()) {
                    visited_.insert(e);
                    return;
                }
            }
        }
        virtual E&    operator*() const { return _Super::operator*(); }
        virtual ItrBase<E>* clone() const { return new _Self(*this); }
    protected:
        virtual bool equal(const ItrBase<E>& o) const { return _Super::equal(o); }
    protected:
        set<E> visited_;
    };     

    // Build the container’s range w/ and w/o repetition
    Itr<E> begin(Container& c, bool noRepeat = false)
    {
        Itr<E> o;
        if (noRepeat) {
            o.itr_ = new ItrNoRepeat(c);
        } else {
            o.itr_ = new ItrAll(c);
        }
        o.itr_->itr_ = c.begin();
        return o;
    }     

    Itr<E> end(Container& c, bool noRepeat = false)
    {
        Itr<E> o;
        if (noRepeat) {
            o.itr_ = new ItrNoRepeat(c);
        } else {
            o.itr_ = new ItrAll(c);
        }
        o.itr_->itr_ = c.end();
        return o;
    }

Related posts:

1. What is software quality?
2. How to make software deterministic

Paul rightly emphasized three trends. The first one is well know: the continuously rising cost of IC designs, about $50M for today’s 45nm node. The second trend is that the fastest growing part of the design cost is software –more than half of the overall cost, Paul even claiming close to 2/3 of the overall cost. The third trend is an increasingly fragmented consumer market: the number of end products goes into the 10’s of billions, but these products are declined in many more different kinds, which means that most of them are shipping in smaller individual volumes.

Source: Morgan Stanley, Economy + Internet Trends, Web 2.0 Summit, San Francisco, Oct 2009.

This is bad news for EDA as we know it: the rising cost of design can no longer be justified if the number of units does not grow fast enough (a $50M chip starts to make sense only if it is produced for 250M units and more). Also EDA has been slow to climb up the food chain and proposes solutions for software design, which dominates the overall chip design cost.

Rising IC design cost and smaller number of units is the call for FPGA to growth even faster. Mobile applications require FPGA to do much better in terms of power consumption, but this is a hot topic (no pun intended) drawing a lot of attention and investment, and some competitive solution will emerge in the next few years. So EDA, which makes its bread and butter on IC design, should better re-align its growth strategy on software, embedded systems, HW/SW co-design, and verification. Else EDA will continue to shrink to only service the few that can still afford chip design.

The end product, as a SoC, is a puzzle where the designer mostly assembles existing cores and IPs, and decides of the tradeoff between the software and hardware parts, based on flexibility and cost factors.

I see two strong needs that EDA could build its growth on. One is functional validation of the whole system –software plus hardware. EDA has started to address the issue, even though it is still short of proposing a scalable and automated environment. To functional validation, I would also add functional flexibility: how much of the behavior can be upgraded thanks to the software part? The other need is a design navigator that would estimate the speed, area, power consumption, and cost of a SoC by exploring alternatives between cores (ARM, MIPS, etc), IPs, FPGA, and software.

Last but not least, the eternal question of an EDA serving a $250B semiconductor industry, but making less than $5B. The time-based license model has only served the interests of the semiconductor companies, to the expenses of R&D investment in EDA. Claiming a lack of innovation in the EDA industry is sometimes fair, but EDA should also innovate in business solutions instead of cannibalizing itself by cutting costs to only survive another quarter. The semiconductor industry needs a healthy EDA if it wants to address the system-level design challenges of the next 10 years. Unless, of course, a new player coming from the software world with the experience of scalable systems signs the death of the EDA industry as we know it.

Related posts:

1. The formal verification market is still untapped

Comparing the yearly fiscal exercises directly would be biased (Xilinx’ fiscal year end on March 31^st, and Altera’s fiscal year on Dec 31^st). Instead we can look at a quarter by quarter comparison, even though that can be too low a level. Better is to look for ttm (trailing twelve months) comparison to smooth out the local variations.

Source: Yahoo! Finance. All figures in thousands.

One can see that Altera’s operating margin is overall better. Also in their respective Q3’09 revenue reports, Xilinx expects its Q4’09 gross margin to improve to 62-63%, and Altera sees his to be 67-68%.  So a 3-4% operating margin gap will remain, which is significant.

On the other hand, Xilinx quotes 3145 full time employees, and Altera 2760. This means that a Xilinx employee brings back revenue about 26% higher than an Altera employee! So it all boils down to the question: how can Xilinx be more cost efficient?

One of the differences is the way software is developed. Altera’s software is mostly done in their technology center of Penang, Malaysia, with a very small core technology group in Toronto, Canada. Xilinx’s software team is mostly in the US, and only 5% of the team is in their R&D facilities in Hyderabad, India. A back-of-the-envelop calculation shows that if Xilinx had the same software team but with a US/India ratio 1/3-2/3, which is a healthy ratio for a company that can leverage its India facility, Xilinx would improve its operating margin by one point.

If you extend the same reasoning to whole R&D –not only software–, then it is clear that Xilinx can get the upper hand. Looking at the R&D job listings, it is also clear that Xilinx is moving into that direction. The question then is whether Xilinx has the structure and the drive to achieve such a transformation successfully.

Related posts:

1. What to read in Xilinx’ and Altera’s third quarter results
2. Who should worry about Xilinx and Oasys partnership?

Defect rate

First I would like to discuss the notion of software reliability and how it evolved over the past 40+ years. A defect causes an invalid behavior of a program with respect to its specification (e.g., incorrect output, performance issue, crash). One of many ways to look at software quality is to estimate its defect rate, i.e., the number of defects per line of code (loc), or more conveniently per 1,000 lines of code (kloc).

The first observation is that the larger the code, the higher its defect rate. It is estimated that the bug rate increases logarithmically with code size.

Source: Program Quality and Programmer Productivity, Capers Jones, IBM 1977

Thus the total number of defects for a specific application can be reduced by the following:

1. Continuous code factorization (direct loc reduction).
2. Use of libraries (which have a reduced bug rate, thanks to the extensive exposure they receive due to their long lifespan and high usage).
3. Increase the expressive power of the programming language (indirect loc reduction).

Since the introduction of FORTRAN in 1957, many languages and operating systems have been created and have grown more powerful and sophisticated. What could be typically coded in 10 klocs of FORTRAN can be coded today with less than 5 klocs of C++, and about 3-4 klocs of Java. Raising the level of abstraction of programming languages helps decreasing the total number of defects because it results in smaller programs with a lower defect rate.

Evidently, testing reduces the defect rate. A software powerhouse like Microsoft reports about 10-20 defects/klocs before QA, and claims that the rate drops to 1/kloc in released code. Looking at long lifespan and very critical code, statistic from the Jet Propulsion Laboratory shows that spacecraft software (which is typically only 20 klocs, and must run without interruption for years) reaches 6-10 defects/klocs after 2-5 years of testing. The code developed for the shuttle program is estimated to have less than 0.1 defect/klocs.

Source: Nikora, Allen P., “Error Discovery Rate by Severity Category and Time to Repair Software Failures for Three JPL Flight Projects”, Software Product Assurance Section, Jet Propulsion Laboratory, November 5, 1991″

Over the past 40 years, independent researches from academia and the private sector have shown that on average an application has a defect rate of 5.5/klocs, regardless of the programming language and the operating system used for development. This looks counterintuitive, since increasing the abstraction level of the programming language reduces the bug rate and the actual size of one specific application. But that progress is neutralized by the ever-increasing size and complexity of the programs, made possible by better software development methodologies and powerful development environments. To put a defect rate of 5.5/kloc in perspective, consider your typical EDA place-and-route product, say 3Mlocs of C/C++, with a likely high turnover rate (i.e., percentage of locs that are modified in every release). You can expect about 16,000 defects…

Test-Driven Design

Now I will present a method that I successfully used for both existing and from-scratch products. It is based on the observation that independently from the quality of the team and the advancement of the tool, the software complexity and the unpredictable evolution of the product makes managing the software quality quite problematic. Think EDA, where customers ask for new capabilities every week and salespeople sell features 6 or 12 months before they are actually developed. It is difficult, if not impossible, to have an upfront, clean, and frozen specification, from which an architecture and a set of APIs can be derived. One needs to change the architecture and the APIs because of new unpredicted features and unforeseen problems, or simply because the software is written in a hurry without the adequate resources –I have no doubt that most readers will agree on that last point. This creates bloated code with a high defect rate, which result in application with a larger number of bugs.

Test-driven design flips the traditional software development scheme upside-down. In most cases, the software development flow consist of (1) specify the requirements in some language (e.g., English, ML, C++ or Java header files), and (2) iterate a code/test loop until the software reaches a point where it is deemed stable enough to go through a full QA regression release process. This often leads to slow iterations between the release team and the R&D team before the release is fully qualified. Also the essence of the original specification may be lost because there is no concrete way (read: operational semantics) to check whether the released product actually meets its intended requirements.

Traditional vs. test-driven software development flow

Contrast this with a test-driven design approach. In that methodology, the tests are written before anything else. The goal is to capture the specification with a set of small (positive and negative) unit tests. Then some code is written and run on the unit tests. Some of the tests fail, which lead to further refinement of both the unit tests and the code. This iteration write-test/code/test converges until one cannot design a new test that would break the code. The next step, QA regression release process, can then be carried on.

A few things are important to recognize in a test-driven software development methodology: (1) the spec is the set of unit tests; (2) therefore the release can be validated as meeting the spec; (3) the testing iteration handled by R&D is closed when the unit tests and the code are fully stable, which leads to fewer iterations between the release and R&D teams; and (4) this methodology does not assume anything about the intrinsic quality of the code and the strength of the development team. Indeed this approach can be used on very badly architected code and still lead to substantial improvements.  Also note that the unit tests can be internal, e.g., written in C++ and providing a self-testing mechanism, or more traditional with external data that are fed to the application.

Case studies

Let me give a few concrete examples. A tool I was in charge of contained some legacy code that performed an essential task in EDA: constant propagation (it consists of propagating logic values through a logic network, following basic computation rules, e.g., NOT(0) = 1, AND(0, 1) = 0, and AND(1, 1) = 1). The computational principles are simple, but a good constant propagation system should be lazy, incremental, support undo, may explain to the user why some constant occurs in some part of the network, etc.  This makes the development of the system much more challenging.

The legacy code produced crashes now and then. It was difficult to read, it contained suspicious piece of code to handle corner cases (e.g., multi-driver nets, user-set constants), and it had a poor testing coverage (<50%). I decided to go for a full rewrite with a clean API, and unit tests were developed together with the new code following a TDD methodology. This resulted in 6267 loc of C++, 40% of which being unit tests (click the screenshot of the C++ unit tests below), made of 1415 asserts. That code was release in May 2007, got 3 reported defects until November 2007, and has been without defect since then.

Another example is a C++ template’ized bitwise four-valued simulator, written to match the Verilog semantics. This was done with 8014 loc of C++, including 40% of unit tests, made of 1015 asserts (click the screenshot below: you can recognize the basic four-valued logic truth tables).  The template was self-tested with three different concrete instances of logic representation (on 2-tuples of bool, on strings made of 32 or 64 characters ’0′, ’1′, ‘x’, and ‘z’, and finally on an actual logic netlist).  No defect was ever found on the semantics.

In both these cases, I had the opportunity of rewriting or starting from scratch. What if one has to improve on an existing system too large to be rewritten?

The third example is about a complex feature (sequential clock gating) that at the time had been released 6 months before. The field complained about inconsistencies and erratic behavior, so I decided to apply a TDD methodology to rectify the code. First hurdle, we established a unit test campaign, which consists of describing the spec in terms of unit tests in plain English and sketches. This produced 49 unit tests, as shown below (click to enlarge).

Second hurdle, we proceeded to translate these informal unit test descriptions into elementary RTL descriptions. The idea was that if the code was compliant to the spec, we could predict exactly which optimized netlist it would produce. Third hurdle, a 3^rd party reviewed these 49 RTL tests, and found that 9 of them were faulty because they did not capture what was specified in the document. Once we fixed these tests came the fourth hurdle: we run the code.

The results were brutal: the code crashed on 3 tests, it synthesized a functionally incorrect netlist in 5 cases, and produced 13 suboptimal results. Overall, 21 failures out of 49 tests, a 43% defect rate! We then went through a 2 weeks iteration of unit test refinement and code fixing with a team that never touched the initial code, to eventually converge on 72 unit tests –many more than we could think of initially– and a usable feature.

Conclusion

Test-driven design (TDD) aims at capturing a spec with unit tests, then have some code successfully running these tests. The unit tests are more important than the code itself –any code that passed the unit tests meets the spec–. TDD initially requires a higher investment: writing unit tests to capture an expected behavior is a complex task, and a 3^rd party review is needed to validate them. But the effort pays off: eventually the set of unit tests becomes the spec, and can even be used as documentation. Running unit tests is fast, so it dramatically reduces the R&D testing time. Also once a code passes a comprehensive set of unit tests, the risk of iterating from QA back to R&D is reduced. Overall, test-driven design increases code correctness and stability dramatically, even in the presence of a deficient architecture and legacy code.

Related posts:

1. API design 101
2. What is software quality?
3. How to make software deterministic

I also worked with already established products, with larger team and millions of lines of already existing code. The typical software management and development project always offers some cumbersome legacy code and API that survive years after years. The reason is not so much that people do not want to fix the problem, but that fixing the problem requires a major product architecture overhaul, which comes to a prohibitive cost. There are striking lessons in failed software architectures, and it all start with API design. I am sharing here my practical experience with C++ projects, but most of these advices also apply to Java.

Why is API so important?

An API can be a company’s greatest asset: it captures communication and exchange of services in an application. A good API will naturally lead to more reuse, simpler code, and lower maintenance cost. If the API is public, a good API will also capture customers. There are examples of Java libraries that failed to be accepted not because they were inefficient, but because they very poorly designed.

An API can also be a company’s greatest liability: once the service has clients, one can no longer change the API!  Suspending or rewriting an API is very pricey in terms of time and money. In the case of a public API, cost also comes in terms of reputation. A public API is forever: there is only one chance to get it right.

What is a good API?

In today’s object-oriented software, writing an API is providing a service. Thus instead of thinking in terms of implementation and efficiency, one must first think in terms of modules and services: determine the usage model; establish the clients’ needs; and anticipate tomorrow’s needs.

Besides being powerful enough to satisfy the requirements, an API should be designed with two principles in mind:

1. Keep it simple! An API must be easy to learn and use, even without documentation. The API must be hard to misuse. Functionality should be easy to explain –if it is hard to name, it is likely a bad function. Use simple, consistent naming, and the code will read like a prose –Java libraries and STL are good inspirations for naming conventions. The API should be as small as possible: you can always add to an API, but you can never remove. A method should not take more than 3-4 parameters –else wrap the parameters in a class that can be augmented later.
2. Keep it abstract! An API must allow extension for future needs. For example, it should not assume anything about the implementation. It should minimize accessibility to implementation-specific details –an API, once public, will be used, and you do not want to expose the ugly details of a database.

In theory, an API should be written before going into some implementation. Gathering requirements is the first step. Requirements must be case-driven, specific, and should be questioned relentlessly until proven to be must-have. The API should then be written in the target language (C++, Java, etc): this will force the development team to make choices, and to keep the API simple and abstract enough –nobody wants to have too much to discuss!  Then the API should be reviewed and made final in a public forum with the two principles above in mind: keep it simple (so that it is easy to support) and abstract (so that it is easy to extend).

An API should be documented, but well-designed APIs are sometimes self-explanatory. An API should answer the following questions about its components.

1. Class: what does an instance of a class represent?  Is that a singleton class?  Is there a factory?  Who owns the memory?
2. Method: what does it do?  What is the contract between the client and the instance?  Is there any precondition and post-condition?  Is there any side effect?
3. Parameters: what do they represent?  Which information do they carry?  Who own them?
4. Exceptions: who throw exceptions?  What do they mean?  What to do when catching one?

API and performances

Bad API decision can limit performances. When designing an API, it is good to consider the following rules.

1. Avoid mutability. If a method returns a mutable instance, that instance needs to be created somewhere, which raises the question of memory ownership. Also mutable classes limit thread-safeness. Use ‘const’ whenever possible.
2. Avoid implicit call to copy and assignment operators. This is a waste of resources if you can use references. Declare these operators ‘explicit’ or ‘private’ to catch any misuse at compile time.
3. A factory is often better than constructors. A factory has full control on how instances are created and when they should be released (shared model, garbage collection, save/restore, caching and disk mirroring, etc). A factory can return an instance of a sub-class.
4. Avoid exposing implementation details. It may prevent later improvements of a database. Never expose data members of a class, always use get/set accessors.
5. Question the thread-safeness of computational-intensive methods. One day the software may run on a grid or in a cloud.
6. Never compromise the rules above for a small runtime or memory improvement. For the vast majority of the applications, going a few percents faster is not worth the maintenance nightmare it can imply.

Final word

A good API is a key to produce smaller and simpler code, which makes the product more stable and easier to maintain. Designing a good API is a collaborative effort, and a formal decision process is needed to freeze an API. A good API is hard to write, get your best people on it. And finally, a public API is forever. May these simple rules guide your next project.

Related posts:

1. Test-driven design, a methodology for low-defect software

• Outsourcing (included in dictionaries in 1979): the procuring of services or products, such as the parts used in manufacturing a motor vehicle, from an outside supplier or manufacturer in order to cut costs.
• Offshoring: relocation by a company of a business process from one country to another –typically an operational process, such as manufacturing, or supporting processes, such as accounting.

In many service and manufacturing industries, outsourcing implies that the 3^rd-party provider is established abroad, where the cost of labor and production is lower, or where the environmental laws and ethic is held to a lower standard.   Signs of the times, outsourcing is often used interchangeably with offshoring.

Massive offshoring started with textile and clothe industry in the late 70’s.  Then came toys, TV, hotlines, help desks, cars and electronics in the 80’s, to be followed by software in the late 90’s.  Over the past 30 years, people have accepted the idea that the products they buy and use in everyday life can be produced and assembled on another continent, where the cost of labor is lower and the labor and business laws are less restrictive –today nobody expects a toy to be produced anywhere but in China.  Soon people will be insensitive to the idea that their software is designed and produced in India or China, especially if it is embedded in ubiquitous hardware like cell phones, game consoles, or digital cameras.  It will even be less of a question with web-based applications, where cloud computing and distributed data centers make physical location irrelevant.

Software offshoring results from a natural evolution of the industry.  Like for so many other industries, complexity required a more organized production process.  Software development evolved from a highly specialized, hand-crafted process, to an application-driven, methodology-centric, maintenance-heavy operation.  The availability of skilled labor and software development methodologies open the door to outsourcing, then offshoring.  For long held as a high-intellectual product that could only be conceived in a handful of countries in the western world, software can now be designed, produced, and maintained in any place that have access to highly educated engineers, with a relatively simple infrastructure –computers and fast internet connections.  One should rejoice to the idea of an industry that can be established anywhere innovation has the opportunity of blossoming, as opposed to a monopoly held by a few companies in a couple of countries.

Indeed, outsourcing of intangible products –e.g., service, consulting, design— and BPO (Business Process Outsourcing) which started in the early 80’s, got a huge boost in the 90’s.  With the tech and telecommunication bubble, massive investments in submarine cables for intercontinental high-bandwidth communication were done.  Running from Europe to India via Egypt, hub centers in Bangalore, New Delhi, Hyderabad, Chennai, Pune, and Mumbai saw their capacity increase dramatically.  Soon real-time and reliable data exchange via the internet made high-tech outsourcing a reality.  After being a BPO bonanza, Bangalore quickly emerged as the Indian Silicon Valley.  Hundreds of software and hardware design companies set foot there, first with help centers and QA engineering, then with HW/SW supporting development teams, to finally complete design and development entities.  Other countries have developed huge HW/SW outsourcing businesses –China, Philippines, and Malaysia, to name a few, as well as some east-Europe countries.

Today, the cost of a software developer in India is about a third to a fourth than in the US –the figure varies depending on the industry, and it becomes cheaper as the experience and complexity requirements decrease.  In China, it will cost about a fifth to a tenth –very dependent on the industry domain, as well as the location in China.  Major US and European high-tech companies like Oracle, ST, Intel, Adobe, SAP, IBM, Microsoft, Google, Yahoo!, have very large R&D campuses in India and China.  For many, their facilities in India are the biggest outside of the US.  For some, most of the R&D growth is seen outside of the US/Europe.  It is not rare to see successful high-tech companies, created in the Silicon Valley 10+ years ago, but with 90% of their R&D today outsourced in Asia.

As a consumer, pretty much nobody complains about outsourcing: in today’s world of rapid consumption of electronic gadgets and complex software, one needs to spin new products at a rapid pace and for an ever more competitive price.  However, offshoring costs jobs at home, which eventually translate to lower disposable incomes and additional social costs, both negatively impacting the local economy.  The creation of wealth in the host country has a side effect though: increasing the disposable income abroad creates new customers for the home business, thus at the end everybody may benefit from it.  This is a more positive scenario, probably true in the long run, but the lag between the disappearance of an activity and its replacement with another comes with a significant social cost.

Also we have seen offshoring displace industries entirely, and the intellectual-content of the displaced industry keeps increasing:  there is virtually no textile industry in Europe and in the US, and UK’s manufacturing industry is trailing in Europe.  The usual response to these displacements is that more lucrative activities replace those that moved abroad.  London has long promoted the dismantlement of its manufacturing industry via offshoring as a chance to move to a service and finance fueled economy, which produces a higher added-value.  But with the recent economic downturn driven by the finance industry, one cannot help though but question the soundness of that claim.

At the end, software outsourcing is here to stay: there is too much to gain for the home companies and the host countries, and the low cost of the infrastructure makes it flexible and easy to extend.  On Sand Hill it is common to hear VCs asking “What is your Indian strategy?”.  Some startups in the Bay Area even start from day one will a full software development team established in India, with just the executives, sales and support located in the US.  Needless to say, these companies could not thrive or even get started without outsourcing part of their software development.  Since they eventually contribute to the high-tech industry, one should endorse the long-term benefit.

Does that mean that being a SW/HW engineer in the Silicon Valley has become a high-risk job?  Software innovation still relies on individuals with bright ideas for technology and products, thus these individuals will always be in high demand.  But it has certainly become much harder for the general-purpose software developer.  The Silicon Valley has benefited from a unique highly-educated engineer pool, entrepreneurs, and VC money.  As long as these three components remain, there is no threat in sight.  But if more and more VCs and entrepreneurs start to establish themselves in India, we will see a very serious competitor to the crown of software kingdom.  Software outsourcing will not kill the Silicon Valley.  Lack of innovation will.

No related posts.