Netrix's devLog

Recently I’ve finished Machine Learning course on Coursera and I think I can recommend it to anyone who wants to start doing something in this field. This area is not something new to me since my specialization on master’s degree was Intelligent Information Systems which covered this area pretty well. By doing this course I wanted to get another point of view on that. Now I can definitely say that prof. Andrew Ng does a good job explaining the material presented on the course. He starts with cost function, gradient descent which are the foundation of Machine Learning. Then he explains such topics as:

• classification (with multiclass classifications),
• logistic regression,
• neural networks with backpropagation,
• bias and variance,
• SVM,
• clustering,
• anomaly detection,
• photo OCR.

Everything is presented with real file examples. Students are required to write some code with the elements explained in each part in Matlab or Octave which are supported by scripts that simplify the way of submitting answers. If there is a one thing I could complain about it would be the out of date scripts that for newer version of Octaves require some patches which can be easily found on course’s wiki.

C++14 introduced new type of templates which are variable templates1. What it means is that it’s now possible to create template of a variable like this:

1
2
3
4

template<class Type>
constexpr Type PI = (Type)3.1415;
 
auto v = PI<double>;

This introduces a variable that can take different values for different template parameter. Main gain comes when such value is specialized like template:

1
2
3
4

template<>
constexpr int PI = 4;  // Because I can
 
auto v = PI<int>;

Of course in previous versions of standard this could be done in a slightly different way by using functions like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

template<class Type>
constexpr Type PI()
{
   return (Type)3.1415;
}
 
template<>
constexpr int PI()
{
   return 4;  // Because I can
}
 
auto v = PI<double>();

The difference in usage is “()” which is used in a function call when using function template version. But there is also another difference which makes variable template unfortunately less useful than function template. Since C++11 it is possible to delete a specialization of function template but it seems it’s not possible to delete specialization of variable template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

template<class Type>
constexpr Type PI()
{
   return (Type)3.1415;
}
 
template<>
constexpr int PI() = delete;
 
template<class Type>
constexpr Type ANSWER = (Type)42;
 
//template<>
//constexpr double ANSWER<double> = delete;   // not possible

But of course no one says that we cannot use both as a syntactic sugar:

1
2

template<class Type>
constexpr Type PI_VAR = PI<Type>();

1. http://en.cppreference.com/w/cpp/language/variable_template [↩]

Two days ago I was working on an old neural network project when some strange bug occured in my code. I was working on debug version of the project with no optimizations (-O0) and then I tried to switch them on to see what will be the gain in performance.

What happened is that it stopped working properly. With no optimizations it worked correctly, giving expected results but with -O2 flag it somehow stopped earlier than expected. After some debugging I’ve managed to find the place where the bug was hidden. This is the place:

1
2
3
4

size_t getOutputLayerSize() const
{
    m_forwardNetwork.getOutputLayerSize();
}

The bug here should be pretty obvious but when running through more code it may well hidden. The obvious thing is that there’s no return statement while the function is defined as returning an unsigned integer value. This code compiles because it may be a valid behaviour in some other cases like following:

1
2
3
4

size_t getOutputLayerSize() const
{
    throw std::runtime_error("");
}

But when nothing is thrown and there is no return statement this is an undefined behaviour and should be avoided.

Why it worked in my case with no optimizations? The m_forwardNetwork.getOutputLayerSize() function call returns an integer value that is saved in eax register. The same register is used in getOutputLayerSize() function to keep the return value which is normally saved by the return statement. With no return statement there is no action but since the value was written in m_forwardNetwork.getOutputLayerSize() it can be used by the caller of getOutputLayerSize() function so it happens to work correctly from caller POV. When optimizations are switched on, the line m_forwardNetwork.getOutputLayerSize() is removed by the compiler because it is not used so there is no function call which means eax register is not written to. In my case it happened that the overal function call return 0 value which wasn’t expected.

My fault was that I haven’t used any warning flags. By adding -Wall compiler starts to emit following warning:

warning: no return statement in function returning non-void [-Wreturn-type]

My advice is to always add -Wall to compilation flags so no error will be omitted. Another advice is to also add a -Werror flag which will force the compiler to fail the compilation if any warning occurs.