Netrix's devLog

Recently I’ve finished Machine Learning course on Coursera and I think I can recommend it to anyone who wants to start doing something in this field. This area is not something new to me since my specialization on master’s degree was Intelligent Information Systems which covered this area pretty well. By doing this course I wanted to get another point of view on that. Now I can definitely say that prof. Andrew Ng does a good job explaining the material presented on the course. He starts with cost function, gradient descent which are the foundation of Machine Learning. Then he explains such topics as:

• classification (with multiclass classifications),
• logistic regression,
• neural networks with backpropagation,
• bias and variance,
• SVM,
• clustering,
• anomaly detection,
• photo OCR.

Everything is presented with real file examples. Students are required to write some code with the elements explained in each part in Matlab or Octave which are supported by scripts that simplify the way of submitting answers. If there is a one thing I could complain about it would be the out of date scripts that for newer version of Octaves require some patches which can be easily found on course’s wiki.

C++14 introduced new type of templates which are variable templates1. What it means is that it’s now possible to create template of a variable like this:

1
2
3
4

template<class Type>
constexpr Type PI = (Type)3.1415;
 
auto v = PI<double>;

This introduces a variable that can take different values for different template parameter. Main gain comes when such value is specialized like template:

1
2
3
4

template<>
constexpr int PI = 4;  // Because I can
 
auto v = PI<int>;

Of course in previous versions of standard this could be done in a slightly different way by using functions like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

template<class Type>
constexpr Type PI()
{
   return (Type)3.1415;
}
 
template<>
constexpr int PI()
{
   return 4;  // Because I can
}
 
auto v = PI<double>();

The difference in usage is “()” which is used in a function call when using function template version. But there is also another difference which makes variable template unfortunately less useful than function template. Since C++11 it is possible to delete a specialization of function template but it seems it’s not possible to delete specialization of variable template:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

template<class Type>
constexpr Type PI()
{
   return (Type)3.1415;
}
 
template<>
constexpr int PI() = delete;
 
template<class Type>
constexpr Type ANSWER = (Type)42;
 
//template<>
//constexpr double ANSWER<double> = delete;   // not possible

But of course no one says that we cannot use both as a syntactic sugar:

1
2

template<class Type>
constexpr Type PI_VAR = PI<Type>();

1. http://en.cppreference.com/w/cpp/language/variable_template [↩]

Two days ago I was working on an old neural network project when some strange bug occured in my code. I was working on debug version of the project with no optimizations (-O0) and then I tried to switch them on to see what will be the gain in performance.

What happened is that it stopped working properly. With no optimizations it worked correctly, giving expected results but with -O2 flag it somehow stopped earlier than expected. After some debugging I’ve managed to find the place where the bug was hidden. This is the place:

1
2
3
4

size_t getOutputLayerSize() const
{
    m_forwardNetwork.getOutputLayerSize();
}

The bug here should be pretty obvious but when running through more code it may well hidden. The obvious thing is that there’s no return statement while the function is defined as returning an unsigned integer value. This code compiles because it may be a valid behaviour in some other cases like following:

1
2
3
4

size_t getOutputLayerSize() const
{
    throw std::runtime_error("");
}

But when nothing is thrown and there is no return statement this is an undefined behaviour and should be avoided.

Why it worked in my case with no optimizations? The m_forwardNetwork.getOutputLayerSize() function call returns an integer value that is saved in eax register. The same register is used in getOutputLayerSize() function to keep the return value which is normally saved by the return statement. With no return statement there is no action but since the value was written in m_forwardNetwork.getOutputLayerSize() it can be used by the caller of getOutputLayerSize() function so it happens to work correctly from caller POV. When optimizations are switched on, the line m_forwardNetwork.getOutputLayerSize() is removed by the compiler because it is not used so there is no function call which means eax register is not written to. In my case it happened that the overal function call return 0 value which wasn’t expected.

My fault was that I haven’t used any warning flags. By adding -Wall compiler starts to emit following warning:

warning: no return statement in function returning non-void [-Wreturn-type]

My advice is to always add -Wall to compilation flags so no error will be omitted. Another advice is to also add a -Werror flag which will force the compiler to fail the compilation if any warning occurs.

C++11 has this new feature that allows programmer to remove:

• automatically generated method or operator,
• overload of free function or method (from implicit conversion)
• specialization of template method.

This can be done by marking any function with = delete keyword and it works like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

void foo(unsigned a)
{}
void foo(int) = delete;                         // deleted function overload for implicit conversion to int
 
template<class T>
void bar(T a) 
{}
template<class T>
void bar<double>(double) = delete;              // deleted template function specialization for double type
 
struct Foo 
{
    Foo& operator=(Foo const&) = delete;        // deleted defaultly generated copying assignment operator
 
    void call(unsigned a) {}
    void call(int a) = delete;                  // deleted method overload for implicit conversion to int
};
 
int main()
{
    foo(1u);
    //foo(2);     // error: use of deleted function 'void foo(int)'
 
    bar(Foo());
    //bar(4.0);   // error: use of deleted function 'void bar(T) [with T = double]'
 
    Foo f;
    f.call(1u);
    //f.call(2);  // error: use of deleted function 'virtual void Foo::call(int)'
 
    Foo ff;
    //ff = f;     // error: use of deleted function 'Foo& Foo::operator=(const Foo&)'
}

It is pretty simple and useful. This allows to remove not wanted specialization from any interface, e.g. overloaded signed value if unsigned and only unsigned value is required. Let consider following class hierarchy:

1
2
3
4
5
6
7
8

struct Foo 
{
    virtual void call(unsigned a) {}
};
 
struct Bar : Foo
{
};

What if it is necessary to remove an overload of call method with int argument from interface of base class? Let see what will happen if different variants of call method are called from base class and derived class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

struct Foo 
{
    void call(unsigned a) {}
    void call(int) = delete;
};
 
struct Bar : Foo
{
};
 
int main()
{
    Bar b;
    b.call(1u);
    //b.call(1);     // error: use of deleted function 'void Foo::call(int)'
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

It works fine. Bar class uses Foo class method declaration so it is not possible to call the call method from both Foo and Bar classes. But what if a programmer would like to have the call method virtual and overridden in derived class?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

struct Foo 
{
    virtual void call(unsigned a) {}
    void call(int) = delete;
};
 
struct Bar : Foo
{
    void call(unsigned a) override {}
};
 
int main()
{
    Bar b;
    b.call(1u);
    b.call(1);       // works since it's not deleted in derived class and can be implicitely casted to unsigned
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

Now it is possible to call the call method from Bar class but not from Foo class. But what if ‘virtual’ is put before void call(int) = delete?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

struct Foo 
{
    virtual void call(unsigned a) {}
    virtual void call(int) = delete;
};
 
struct Bar : Foo
{
    void call(unsigned a) override {}
};
 
int main()
{
    Bar b;
    b.call(1u);
    b.call(1);       // works since it's not deleted in derived class and can be implicitely casted to unsigned
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

It seems it does not matter if function is virtual or not. But there is another keyword that works with inheritence – final which means that virtual method marked with this keyword cannot be overriden in derived class.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

struct Foo 
{
    virtual void call(unsigned a) {}
    virtual void call(int) final = delete;
};
 
struct Bar : Foo
{
    void call(unsigned a) override {}
};
 
int main()
{
    Bar b;
    b.call(1u);
    b.call(1);       // works since it's not deleted in derived class and can be implicitely casted to unsigned
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

It does not matter either if method is final or not. So one and only option is just to declare the call method with int argument as deleted in derived class which should be a first choice of those considerations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

struct Foo 
{
    virtual void call(unsigned a) {}
    virtual void call(int) final = delete;
};
 
struct Bar : Foo
{
    void call(unsigned a) override {}
    //void call(int) = delete;           // error: use of deleted function 'virtual void Bar::call(int)'
};
 
int main()
{
    Bar b;
    b.call(1u);
    b.call(1);       // works since it's not deleted in derived class and can be implicitely casted to unsigned
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

Ooops. Method void call(int) is marked as final in base class so it is not even possible to declare it as deleted in derived class. The one and only valid example that works as it supposed to from the beginning look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

struct Foo 
{
    virtual void call(unsigned a) {}
    void call(int) = delete;
};
 
struct Bar : Foo
{
    void call(unsigned a) override {}    // If we want to override call()
    using Foo::call;                     // if we want to use Foo::call()
    void call(int) = delete;
};
 
int main()
{
    Bar b;
    b.call(1u);
    //b.call(1);     // error: use of deleted function 'void Bar::call(int)'
    Foo & f = b;
    f.call(1u);
    //f.call(1);     // error: use of deleted function 'void Foo::call(int)'
}

In conclusion it is not possible to declare the method overload as deleted in base class so that it will affect derived class if base method is marked as virtual and overriden in derived class. This concerns mostly the interfaces with methods that may accept some unsigned key value while there exists some signed key type, that should not be passed to considered methods.

—
All examples were compiled with GCC 5.2 with –std=c++14 flag.

Old C++ and C for a long time allow creating functions with unknown number of arguments. Most common example of such function is printf which prints text with formatted variables. It gets format as first argument and values to be printed in that format in other arguments. Following code shows its declaration:

int printf(const char * restrict format, ... );

Processing such argument list is done in runtime and it utilizes special macros or types from following list1:

• va_start – enable access to variadic function arguments,
• va_arg – accesses the next variadic function argument,
• va_copy – makes a copy of the variadic function arguments,
• va_end – ends traversal of the variadic function arguments,
• va_list – holds the information needed by va_start, va_arg, va_end, and va_copy.

Following example presents simple usage of such function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

#include <iostream>
#include <cstdarg>
 
double sum(int count, ...) 
{
    double sum = 0.0;
    va_list args;
    va_start(args, count);
    for (int i = 0; i < count; ++i) 
    {
        sum += va_arg(args, double);
    }
    va_end(args);
    return sum;
}
 
int main() 
{
    std::cout << sum(4, 25.0, 27.3, 26.9, 25.7);
}

Firstly va_start binds arguments passed to the function by elipsis (…) to variable args of type va_list. It takes instance of such type and number of arguments it want to access. Then all arguments are accessed subsequently by macro va_arg which takes list as an argument and returned type. At the end of processing arguments list should be unbound with macro va_end. To process same list of arguments more than once it can be copied to another list right after processing is started with macro va_copy (see2 for more).

Of course presented code is just an example. Real implementation of such function should utilize std::accumulate. Also this code shows some pitfalls of variadic arguments – number of arguments must be passed as one of firsts arguments and may not match the exact number of arguments passed to the function, also types can be different than expected in method.

1. http://en.cppreference.com/w/cpp/language/variadic_arguments [↩]
2. http://en.cppreference.com/w/cpp/language/variadic_arguments [↩]