How to enable Halide backend for improve efficiency

Introduction

This tutorial guidelines how to run your models in OpenCV deep learning module using Halide language backend. Halide is an open-source project that let us write image processing algorithms in well-readable format, schedule computations according to specific device and evaluate it with a quite good efficiency.

An official website of the Halide project: http://halide-lang.org/.

An up to date efficiency comparison: https://github.com/opencv/opencv/wiki/DNN-Efficiency

Requirements

LLVM compiler

Note

LLVM compilation might take a long time.

• Download LLVM source code from http://releases.llvm.org/4.0.0/llvm-4.0.0.src.tar.xz. Unpack it. Let llvm_root is a root directory of source code.
• Create directory llvm_root/tools/clang
• Download Clang with the same version as LLVM. In our case it will be from http://releases.llvm.org/4.0.0/cfe-4.0.0.src.tar.xz. Unpack it into llvm_root/tools/clang. Note that it should be a root for Clang source code.
• Build LLVM on Linux

  cd llvm_root
  mkdir build && cd build
  cmake -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_TARGETS_TO_BUILD="X86" -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=Release ..
  make -j4

• Build LLVM on Windows (Developer Command Prompt)

  mkdir \\path-to-llvm-build\\ && cd \\path-to-llvm-build\\
  cmake.exe -DLLVM_ENABLE_TERMINFO=OFF -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=\\path-to-llvm-install\\ -G "Visual Studio 14 Win64" \\path-to-llvm-src\\
  MSBuild.exe /m:4 /t:Build /p:Configuration=Release .\\INSTALL.vcxproj

Note

\\path-to-llvm-build\\ and \\path-to-llvm-install\\ are different directories.

Halide language.

• Download source code from GitHub repository, https://github.com/halide/Halide or using git. The root directory will be a halide_root.

  git clone https://github.com/halide/Halide.git

• Build Halide on Linux

  cd halide_root
  mkdir build && cd build
  cmake -DLLVM_DIR=llvm_root/build/lib/cmake/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_VERSION=40 -DWITH_TESTS=OFF -DWITH_APPS=OFF -DWITH_TUTORIALS=OFF ..
  make -j4

• Build Halide on Windows (Developer Command Prompt)

  cd halide_root
  mkdir build && cd build
  cmake.exe -DLLVM_DIR=\\path-to-llvm-install\\lib\\cmake\\llvm -DLLVM_VERSION=40 -DWITH_TESTS=OFF -DWITH_APPS=OFF -DWITH_TUTORIALS=OFF -DCMAKE_BUILD_TYPE=Release -G "Visual Studio 14 Win64" ..
  MSBuild.exe /m:4 /t:Build /p:Configuration=Release .\\ALL_BUILD.vcxproj

Build OpenCV with Halide backend

When you build OpenCV add the following configuration flags:

• ENABLE_CXX11 - enable C++11 standard
• WITH_HALIDE - enable Halide linkage
• HALIDE_ROOT_DIR - path to Halide build directory

Sample

// Sample of using Halide backend in OpenCV deep learning module.
// Based on caffe_googlenet.cpp.

#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
using namespace cv;
using namespace cv::dnn;

#include <fstream>
#include <iostream>
#include <cstdlib>

/* Find best class for the blob (i. e. class with maximal probability) */
static void getMaxClass(const Mat &probBlob, int *classId, double *classProb)
{
    Mat probMat = probBlob.reshape(1, 1); //reshape the blob to 1x1000 matrix
    Point classNumber;

    minMaxLoc(probMat, NULL, classProb, NULL, &classNumber);
    *classId = classNumber.x;
}

static std::vector<std::string> readClassNames(const char *filename = "synset_words.txt")
{
    std::vector<std::string> classNames;

    std::ifstream fp(filename);
    if (!fp.is_open())
    {
        std::cerr << "File with classes labels not found: " << filename << std::endl;
        exit(-1);
    }

    std::string name;
    while (!fp.eof())
    {
        std::getline(fp, name);
        if (name.length())
            classNames.push_back( name.substr(name.find(' ')+1) );
    }

    fp.close();
    return classNames;
}

int main(int argc, char **argv)
{
    std::string modelTxt = "train_val.prototxt";
    std::string modelBin = "squeezenet_v1.1.caffemodel";
    std::string imageFile = (argc > 1) ? argv[1] : "space_shuttle.jpg";

    Net net = dnn::readNetFromCaffe(modelTxt, modelBin);

    if (net.empty())
    {
        std::cerr << "Can't load network by using the following files: " << std::endl;
        std::cerr << "prototxt:   " << modelTxt << std::endl;
        std::cerr << "caffemodel: " << modelBin << std::endl;
        std::cerr << "SqueezeNet v1.1 can be downloaded from:" << std::endl;
        std::cerr << "https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1" << std::endl;
        exit(-1);
    }

    Mat img = imread(imageFile);
    if (img.empty())
    {
        std::cerr << "Can't read image from the file: " << imageFile << std::endl;
        exit(-1);
    }
    if (img.channels() != 3)
    {
        std::cerr << "Image " << imageFile << " isn't 3-channel" << std::endl;
        exit(-1);
    }

    resize(img, img, Size(227, 227));                // SqueezeNet v1.1 predict class by 3x227x227 input image.
    Mat inputBlob = blobFromImage(img, 1.0, Size(), Scalar(), false);  // Convert Mat to 4-dimensional batch.

    net.setInput(inputBlob);                         // Set the network input.

    net.setPreferableBackend(DNN_BACKEND_HALIDE);    // Tell engine to use Halide where it possible.

    Mat prob = net.forward("prob");                  // Compute output.

    int classId;
    double classProb;
    getMaxClass(prob, &classId, &classProb);         // Find the best class.

    std::vector<std::string> classNames = readClassNames();
    std::cout << "Best class: #" << classId << " '" << classNames.at(classId) << "'" << std::endl;
    std::cout << "Probability: " << classProb * 100 << "%" << std::endl;

    return 0;
} //main

Explanation

Download Caffe model from SqueezeNet repository: train_val.prototxt and squeezenet_v1.1.caffemodel.

Also you need file with names of ILSVRC2012 classes: synset_words.txt.

Put these files into working dir of this program example.

1. Read and initialize network using path to .prototxt and .caffemodel files

       Net net = dnn::readNetFromCaffe(modelTxt, modelBin);

2. Check that network was read successfully

       if (net.empty())
       {
           std::cerr << "Can't load network by using the following files: " << std::endl;
           std::cerr << "prototxt:   " << modelTxt << std::endl;
           std::cerr << "caffemodel: " << modelBin << std::endl;
           std::cerr << "SqueezeNet v1.1 can be downloaded from:" << std::endl;
           std::cerr << "https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1" << std::endl;
           exit(-1);
       }

3. Read input image and convert to the 4-dimensional blob, acceptable by SqueezeNet v1.1

       Mat img = imread(imageFile);
       if (img.empty())
       {
           std::cerr << "Can't read image from the file: " << imageFile << std::endl;
           exit(-1);
       }
       if (img.channels() != 3)
       {
           std::cerr << "Image " << imageFile << " isn't 3-channel" << std::endl;
           exit(-1);
       }
   
       resize(img, img, Size(227, 227));                // SqueezeNet v1.1 predict class by 3x227x227 input image.
       Mat inputBlob = blobFromImage(img, 1.0, Size(), Scalar(), false);  // Convert Mat to 4-dimensional batch.

4. Pass the blob to the network

       net.setInput(inputBlob);                         // Set the network input.

5. Enable Halide backend for layers where it is implemented

       net.setPreferableBackend(DNN_BACKEND_HALIDE);    // Tell engine to use Halide where it possible.

6. Make forward pass

       Mat prob = net.forward("prob");                  // Compute output.

   Remember that the first forward pass after initialization require quite more time that the next ones. It's because of runtime compilation of Halide pipelines at the first invocation.
7. Determine the best class

       int classId;
       double classProb;
       getMaxClass(prob, &classId, &classProb);         // Find the best class.

8. Print results

       std::vector<std::string> classNames = readClassNames();
       std::cout << "Best class: #" << classId << " '" << classNames.at(classId) << "'" << std::endl;
       std::cout << "Probability: " << classProb * 100 << "%" << std::endl;

   For our image we get:

Best class: #812 'space shuttle'

Probability: 97.9812%