\chapter{Experiments}\label{chapter5}

In this chapter, we introduce two-dimensional grammars and show how they can be used to create large instances of SMAF that correspond to Schlesinger's upper bound of a binary max-sum problem. After we describe how the instances are created, we use our algorithm to minimize the given function and compare our result with the optimal value. We also present meaningful modifications of the presented instances, where the recalculation of optimum after a small change is used. Finally, we analyse the initial setting of $\epsilon$ and the minimization process.

\section{Two-dimensional Grammars}\label{twod_gram_section_82}

To create large instances of convex piecewise-affine functions, we will use the notion of two-dimensional grammars. Notice that the notion has changed its meaning recently -- it is nowadays viewed as a generalisation of the "classical" grammars, which is a set of production rules for creation of string of characters. This is described in detail in \cite{prusa2004}, however we will use the "older" meaning defined in \cite{schlesinger1976syntactic}.

In this notion, the pixels of images are assigned labels and there are rules on which labels could be in adjacent pixels. Formally, we will consider only grey-scale images, which will be viewed as matrices of size $h \times w$ with values from the interval $[0,1]$.

\begin{definition}
A {\em two-dimensional grammar} is a binary max-sum problem $(G, X, \bm{g})$ and a function $c : X \to \{0, 1\}$, where
\begin{itemize}
\item $G = (T, E)$ is a grid graph, i.e.
\begin{subequations}\label{grammar_def_54}
\begin{align}
T & = [h] \times [w], \\
E & = \{ \{(i,j), (i',j') \} \mid i,i' \in [h], j,j' \in [w], \abs{i-i'} + \abs{j-j'} = 1\},
\end{align}
\end{subequations}
\item $g_{t}(x) = 0$ for all $t \in T$ and $x \in X$,
\item $g_{t,t'}(x,x') \in \{0, -\infty\}$ for all $\{t,t'\} \in E$ and $x, x' \in X$.
\end{itemize}
\end{definition}

The function $c$ in the definition above can be interpreted as a function that assigns each label either black or white colour. For the purposes of the next sections, assume that black pixels have value 1 and white ones have value 0.

Notice that for the above mentioned binary max-sum problem, it holds for all labellings $\bm{x} \in X^T$ that $F(\bm{x}|\bm{g}) \in \{0,-\infty\}$.

We say that a black and white image $\mathbf{O} \in \{0, 1\}^{[h] \times [w]}$ is {\em generated} by a given grammar if there exists a labelling $\bm{x} \in X^T$ such that $o_{i,j} = c(x_{(i,j)})$ for all $(i,j) \in T$ and $F(\bm{x}|\bm{g}) = 0$.

\subsubsection{Nearest Image Generated by Given Grammar}  
    
If we are given an arbitrary grey-scale image $\mathbf{O} \in [0,1]^{[h] \times [w]}$ and a two-dimensional grammar, then we can formulate the task of looking for the nearest image $\mathbf{O}^*$ of the same size such that $\mathbf{O}^*$ is generated by the given grammar and it is the "closest" one to $\mathbf{O}$ in means of minimizing
\begin{equation}\label{closest_image_measure}
\sum_{i \in [h]}\sum_{j \in [w]} \abs{o^*_{i,j} - o_{i,j}}.
\end{equation}

As already shown in \cite{4204160}, it is possible to express this problem as a binary max-sum problem. The corresponding graph $G$ and the binary functions $g_{t,t'}$ will be the same as in the grammar but the unary functions are given as
\begin{equation}\label{def_of_unary_fcns_22}
g_{(i,j)}(x) = 1-\abs{c(x) - o_{i,j}}
\end{equation}
for all $(i,j) \in T$.

\subsection{Examples of Two-dimensional Grammars}

In this section, we introduce four grammars -- {\em lines, rectangles, pi} and {\em curves}. On the {\em lines} grammar, we show in detail how to define the problem of nearest image generated by the grammar.

\subsubsection{Lines Grammar}

We define a grammar called {\em lines} for images whose matrix contains only vertical and horizontal lines that stretch from top to bottom or from the left border to the right border. More formally, we would like to say that an image $\mathbf{O}$ with size $h \times w$ is generated by the grammar if there exist sets $U \subseteq [h]$ and $V \subseteq [w]$ such that
\begin{equation}
o_{i,j} = \iverson{i \in U \vee j \in V}, \quad \forall (i,j) \in [h] \times [w].
\end{equation}
The set $U$ (resp. $V$) in the previous definition can be viewed as the indices of rows (resp. columns) that contain the line. For example, the images shown in Figures \ref{fig:element1_lines} and \ref{fig:element2_lines} are generated by this grammar, whereas the images in the Figures \ref{fig:element3_lines} and \ref{fig:element4_lines} are not.

\fboxsep=0mm%padding thickness
\fboxrule=1pt%border thickness

\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{0.2\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines1.png}}
            \caption{}   
            \label{fig:element1_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.2\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines2.png}}
            \caption{}   
            \label{fig:element2_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.2\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines3.png}}
            \caption{}    
            \label{fig:element3_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.2\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines4.png}}
            \caption{}
            \label{fig:element4_lines}
        \end{subfigure}
        \caption{Examples of images, first two are generated by the {\em lines} grammar.}
        \label{fig:examples_line_grammar}
    \end{figure*}

There will be four labels, $X = \{H, V, P, I \}$ and we should find a labelling $\bm{x}~\in~X^{[h] \times [w]}$ that assigns each pixel a label from the set $X$. The meaning of the labels is as follows. If a vertical and a horizontal line pass through a pixel, then it should have the label $I$ (intersect). If only horizontal (resp. vertical) line passes through a pixel, then it should have label $H$ (resp. $V$). If no line passes through it, then it is marked with $P$ (empty). The labelled pixels of an image generated by the grammar are shown in Figure \ref{figure_labels_lines}.

The colour assigned to label $P$ is white and the colours of the other labels will be black, i.e. $c(x) = \iverson{x \neq P}$. The values of the unary functions are given by equation \eqref{def_of_unary_fcns_22}.

Now, we only need to properly define the binary functions $g_{t,t'}$ between the neighbouring pixels so that the result is generated by the {\em lines} grammar. That can be done by easy reasoning based on which labels could be neighbours -- for example, above label $V$, there could be label $I$ or another label $V$, but there cannot be label $H$ or $P$. The allowed neighbouring relations are in Tables \ref{table_with_allowed_labels_vertical} and \ref{table_with_allowed_labels_horizontal}. Using these tables, we can simply forbid certain pairs of labels to neighbour in a certain direction by setting the $g_{t,t'}$ values to either 0 (if the pair is allowed) or $-\infty$ (if the pair is not allowed), formally
\[
  g_{t,t'}(x_t,x_{t'}) =
  \begin{cases}
                                   0 & \text{if the pair $x_t, x_{t'}$ is allowed in the direction given by $t, t'$} \\
                                   - \infty & \text{otherwise} 
  \end{cases}.
\]

\begin{table}[h]
\centering
\begin{tabular}{cc|cccc}
    & &\multicolumn{4}{c}{Upper label $x_{(i,j)}$}\\
    & &P&H&V&I\\
    \cline{2-6}
    \multirow{4}{*}{Lower label $x_{(i+1,j)}$}&P& \checkmark & \checkmark & \xmark & \xmark \\
    &H&\checkmark & \checkmark & \xmark & \xmark \\
    &V&\xmark &\xmark &\checkmark & \checkmark \\
    &I&\xmark &\xmark &\checkmark & \checkmark \\
\end{tabular}
\caption{Vertical relations between labels in {\em lines} grammar.}
\label{table_with_allowed_labels_vertical}
\end{table}

\begin{table}[h]
\centering
\begin{tabular}{cc|cccc}
    & &\multicolumn{4}{c}{Right label $x_{(i,j+1)}$}\\
    & &P&H&V&I\\
    \cline{2-6}
    \multirow{4}{*}{Left label $x_{(i,j)}$}&P& \checkmark & \xmark & \checkmark & \xmark \\
    &H&\xmark & \checkmark & \xmark & \checkmark \\
    &V&\checkmark &\xmark &\checkmark & \xmark \\
    &I&\xmark &\checkmark &\xmark & \checkmark \\
\end{tabular}
\caption{Horizontal relations between labels in {\em lines} grammar.}
\label{table_with_allowed_labels_horizontal}
\end{table}

\iffalse
\begin{table}
\centering
\label{bigtablewithallowed}
\caption{Allowed pairs of labels in the {\em lines} grammar}
\begin{tabular}{cc|cccc}
    & &\multicolumn{4}{c}{Upper label $l_{(i,j)}$}\\
    & &P&H&V&I\\
    \cline{2-6}
    \multirow{4}{*}{Lower label $l_{(i+1,j)}$}&P& \checkmark & \checkmark & \xmark & \xmark \\
    &H&\checkmark & \checkmark & \xmark & \xmark \\
    &V&\xmark &\xmark &\checkmark & \checkmark \\
    &I&\xmark &\xmark &\checkmark & \checkmark \\	
    \label{table_with_allowed_labels_vertical}
\end{tabular}
\subcaption{Vertical relations between labels in lines grammar}
\bigskip
\begin{tabular}{cc|cccc}
    & &\multicolumn{4}{c}{Right label $l_{(i,j+1)}$}\\
    & &P&H&V&I\\
    \cline{2-6}
    \multirow{4}{*}{Left label $l_{(i,j)}$}&P& \checkmark & \xmark & \checkmark & \xmark \\
    &H&\xmark & \checkmark & \xmark & \checkmark \\
    &V&\checkmark &\xmark &\checkmark & \xmark \\
    &I&\xmark &\checkmark &\xmark & \checkmark \\
\label{table_with_allowed_labels_horizontal}
\end{tabular}
\subcaption{Horizontal relations between labels in {\em lines} grammar}
\end{table}
\fi

Finally, we managed to transform the problem of searching the "closest" image generated by the {\em lines} grammar into an equivalent problem formalized as a binary max-sum problem. We can similarly define also other problems, resp. grammars.

\expandafter\def\csname box0\endcsname#1#2{}
\expandafter\def\csname box1\endcsname#1#2{\fill[black] (#1-1, -#2+1) rectangle ++(1, -1);}

\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{\textwidth}
            \centering
\begin{tikzpicture}[scale=0.8]
  \def\BITARRAY{
    {0,1,0,0,0,1,1},
    {0,1,0,0,0,1,1},
    {0,1,0,0,0,1,1},
    {1,1,1,1,1,1,1},
    {1,1,1,1,1,1,1},
    {0,1,0,0,0,1,1}%
  }

    \foreach \row [count=\y] in \BITARRAY {
      \foreach \cell [count=\x] in \row {
        \csname box\cell\endcsname{\x}{\y}
        \pgfextra{%
          \global\let\maxx\x
          \global\let\maxy\y
        }%
      }
    }
  \draw[thin,gray] (0, 0) grid[step=1] (\maxx, -\maxy);
  
  \foreach \i in {0,2,3,4}
  {
  \node[black] at (0.5+\i, -0.5) {P};
  \node[black] at (0.5+\i, -1.5) {P};
  \node[black] at (0.5+\i, -2.5) {P};
  \node[white] at (0.5+\i, -3.5) {H};
  \node[white] at (0.5+\i, -4.5) {H};
  \node[black] at (0.5+\i, -5.5) {P};
  }
  
  \foreach \i in {0,4,5}
  {
  \node[white] at (1.5+\i, -0.5) {V};
  \node[white] at (1.5+\i, -1.5) {V};
  \node[white] at (1.5+\i, -2.5) {V};
  \node[white] at (1.5+\i, -3.5) {I};
  \node[white] at (1.5+\i, -4.5) {I};
  \node[white] at (1.5+\i, -5.5) {V};
  }
  
\end{tikzpicture}
\caption{Example of an image generated by  {\em lines} grammar.}
\label{figure_labels_lines}
\vspace{1cm}
        \end{subfigure}
        
        \begin{subfigure}[b]{\textwidth}  
            \centering
\begin{tikzpicture}[ every node/.style={scale=0.9}]
  \def\BITARRAY{
    {0,0,0,0,0,0,0},
    {0,1,1,1,1,1,0},
    {0,1,1,1,1,1,0},
    {0,1,1,1,1,1,0},
    {0,1,1,1,1,1,0},
    {0,0,0,0,0,0,0}%
  }

    \foreach \row [count=\y] in \BITARRAY {
      \foreach \cell [count=\x] in \row {
        \csname box\cell\endcsname{\x}{\y}
        \pgfextra{%
          \global\let\maxx\x
          \global\let\maxy\y
        }%
      }
    }
  \draw[thin,gray] (0, 0) grid[step=1] (\maxx, -\maxy);
  
  \foreach \i in {0,1,2,3,4,5}
  {
  \foreach \j in {0,6}
  {
  \node[black] at (0.5+\j, -0.5-\i) {P};
  }
  }
  
  \foreach \i in {0,5}
  {
  \foreach \j in {1,2,3,4,5}
  {
  \node[black] at (0.5+\j, -0.5-\i) {P};
  }
  }
  
  
  \foreach \i in {2,3,4}
  {
  \node[white] at (0.5+\i, -1.5) {UpB};
  \node[white] at (0.5+\i, -2.5) {I};
  \node[white] at (0.5+\i, -3.5) {I};
  \node[white] at (0.5+\i, -4.5) {LoB};
  }
  
  \node[white] at (1.5, -1.5) {LeUp};
  \node[white] at (1.5, -2.5) {LeB};
  \node[white] at (1.5, -3.5) {LeB};
  \node[white] at (1.5, -4.5) {LeLo};
  
  
  \node[white] at (5.5, -1.5) {RiUp};
  \node[white] at (5.5, -2.5) {RiB};
  \node[white] at (5.5, -3.5) {RiB};
  \node[white] at (5.5, -4.5) {RiLo};
  
\end{tikzpicture}
\caption{Example of an image generated by  {\em rectangles} grammar.}
\label{example_rectangular_with_labels}
\vspace{1cm}
        \end{subfigure}
        
        \begin{subfigure}[b]{\textwidth}   
            \centering
\begin{tikzpicture}[scale=0.8]
  \def\BITARRAY{
    {0,0,0,0,0,0,0,0,0,0},
    {1,1,1,0,0,1,1,1,1,0},
    {0,0,1,0,0,1,0,0,1,0},
    {0,0,1,0,0,1,1,1,1,0},
    {0,0,1,1,0,0,0,0,0,0},
    {0,0,0,1,1,0,0,0,1,1},
    {0,0,0,0,1,0,0,1,1,0}%
  }

    \foreach \row [count=\y] in \BITARRAY {
      \foreach \cell [count=\x] in \row {
        \csname box\cell\endcsname{\x}{\y}
        \pgfextra{%
          \global\let\maxx\x
          \global\let\maxy\y
        }%
      }
    }
  \draw[thin,gray] (0, 0) grid[step=1] (\maxx, -\maxy);
  
  \foreach \i in {0,1,2,3,4,5,6}
  {
  \foreach \j in {0,1,2,3,4,5,6,7,8,9}
  {
  \node[black] at (0.5+\j, -0.5-\i) {P};
  }
  }
  
  
  \node[white] at (0.5, -1.5) {H};
  \node[white] at (1.5, -1.5) {H};
  \node[white] at (2.5, -1.5) {B};
  \node[white] at (2.5, -2.5) {V};
  \node[white] at (2.5, -3.5) {V};
  \node[white] at (2.5, -4.5) {C};
  \node[white] at (3.5, -4.5) {B};
  \node[white] at (3.5, -5.5) {C};
  \node[white] at (4.5, -5.5) {B};
  \node[white] at (4.5, -6.5) {V};
  
  \node[white] at (5.5, -1.5) {A};
  \node[white] at (5.5, -2.5) {V};
  \node[white] at (5.5, -3.5) {C};
  \node[white] at (6.5, -3.5) {H};
  \node[white] at (7.5, -3.5) {H};
  \node[white] at (8.5, -3.5) {D};
  \node[white] at (8.5, -2.5) {V};
  \node[white] at (8.5, -1.5) {B};
  \node[white] at (7.5, -1.5) {H};
  \node[white] at (6.5, -1.5) {H};
  
  \node[white] at (7.5, -6.5) {A};
    \node[white] at (8.5, -6.5) {D};
      \node[white] at (8.5, -5.5) {A};
        \node[white] at (9.5, -5.5) {H};
  
\end{tikzpicture}
\caption{Example of an image generated by {\em curve} grammar.}
\label{example_curve_with_labels}
        \end{subfigure}
        \caption{Examples of labelled images generated by various grammars.}
        \label{fig:labelled_images_grammars_examples_32}
    \end{figure*}

\subsubsection{Rectangles Grammar}

In the {\em rectangles} grammar, we allow black shapes of rectangular form which are not allowed to overlap or touch each other. This grammar is formalized using 10 labels -- empty space (P), left upper corner of a rectangle (LeUp), left lower corner (LeLo), right upper (RiUp), right lower (RiLo), upper line boundary of the rectangle (UpB), lower boundary (LoB), left boundary (LeB), right boundary (RiB) and the inside (I) of the rectangle.

An example of a rectangle with labelled pixels is in Figure \ref{example_rectangular_with_labels}. We do not explicitly write the colouring function $c$ or all the allowed options for vertical or horizontal neighbouring because all of the allowed options are in Figure \ref{example_rectangular_with_labels}, where also the colours of labels can be seen. An example of an image generated by this grammar is in Figure \ref{fig:rect_image}.

\subsubsection{Pi Grammar}

The {\em pi} grammar is similar as {\em rectangles}, except that the colours corresponding to the inner pixels of a rectangle (label I) and the lower boundary (label LoB) are white. This defines a grammar that contains images in which there are symbols that resemble the shape of letter $\Pi$. Figure \ref{fig:rus_p_image} is an example of an image generated by this grammar.

\subsubsection{Curve Grammar}

The last grammar that will be introduced is the {\em curve} grammar, consisting of non-intersecting curves, where a curve is a sequence of pixels so that the next one is in the 4-neighbourhood of the first one. A line can also begin or end at the boundary of the image and an example of an image generated by this grammar is in Figure \ref{fig:curve_image}. Additionally, in the 4-neighbourhood of a pixel that is black, there can be at most 2 other black pixels -- one that is "previous" in the curve sequence and one that is its "successor" in the line sequence. If the line begins at the boundary, there is only one neighbouring black pixel.

To define this grammar, we used 6 labels -- horizontal part of line (label H), vertical part of line (label V), bend from lower vertical to right horizontal (label A), bend from lower vertical to left horizontal (label B), bend from upper vertical to right horizontal (label C), bend from upper vertical to left horizontal (label D). All the allowed options are shown in Figure \ref{example_curve_with_labels}. Notice that label combination AD is allowed in both horizontally and vertically neighbouring pixels to allow zig-zag shape. Similarly, BC (vertically) and CB (horizontally) combinations are allowed for the other direction of zig-zag.


\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rectangles_example.png}}
            \caption{Image generated by {\em rectangles} grammar}   
            \label{fig:rect_image}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russian_p_example.png}}
            \caption{Image generated by {\em pi} grammar}   
            \label{fig:rus_p_image}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{curve_example.png}}
            \caption{Image generated by {\em curve} grammar}    
            \label{fig:curve_image}
        \end{subfigure}
        \caption{Examples of images generated by various grammars.}
        \label{fig:examples_other_grammars}
    \end{figure*}

\section{Tested Instances}

To obtain large instances of SMAF, we first generated baseline images that were generated by one of the previously defined grammars, then independent Gaussian noise was added to the pixels in the images to obtain the input image $\mathbf{O}$. After that, we transformed the problem of searching the "closest" image generated by the grammar to the binary max-sum problem, basically in the same way as stated in the beginning of Section \ref{twod_gram_section_82}. Finally, we created the SMAF that corresponds to the height minimization of the binary max-sum problem defined by the image, which is the process defined in Algorithm \ref{transformationalg_summax}. To also create "harder" inputs, we sometimes used baseline images that are not generated by the given grammar.

The Gaussian noise had zero mean and we tried three settings concerning the standard deviation of the noise -- we used low noise with $\sigma = 0.12$, medium noise with $\sigma = 0.4$ and high noise with $\sigma = 1.2$. If any value of a pixel got out of the $[0,1]$ interval, its value was trimmed to the nearer border.

\begin{table}
\centering
%\hspace{-0.7cm}
 %\vspace{-1cm}
    \begin{tabular}{l|rr|rrr}
    \multirow{2}{*}{\textbf{Instance}} & \multicolumn{2}{c}{\textbf{LP parameters}} \vline & \multicolumn{3}{c}{\textbf{SMAF parameters}} \\  
   & variables  & constraints  & clusters & subfunctions  & dimension \\     \hline
\texttt{lines200} & 756400 & 796800 & 119600 & 796800 & 636800 \\
\texttt{lines100} & 188200 & 198400 & 29800 & 198400 & 158400 \\
\texttt{lines50diamond} & 46600 & 49200 & 7400 & 49200 & 39200 \\ \hline
\texttt{rect100round} & 425800 & 416800 & 29800 & 416800 & 396000 \\
\texttt{rect100sharp} & 425800 & 416800 & 29800 & 416800 & 396000 \\ \hline
\texttt{pi50prec} & 105400 & 103400 & 7400 & 103400 & 98000 \\
\texttt{pi50rough} & 105400 & 103400 & 7400 & 103400 & 98000 \\ \hline
\texttt{curve500} & 7735000 & 8736000 & 749000 & 8736000 & 6986000 
     \end{tabular}
    \caption{Sizes of the individual tested instances.}
    \label{table_comparison_problem_sizes}
\end{table}

We used 8 different baseline images, to which different amounts of noise were added, which resulted in 24 instances of SMAF in total. Table \ref{table_comparison_problem_sizes} contains the parameters of the instances which define their size, i.e. both the parameters of the corresponding upper bound minimizing LP in the form of equation \eqref{dual} and the parameters corresponding to the SMAF that is minimized. Note that these parameters do not depend on the amount of noise in the image but only on the used grammar (i.e. the amount of labels and the rules on neighbouring labels) and the image size.

The first part of the name of an instance consists of the name of the grammar by which the resulting image should be generated (i.e. $\mathtt{lines}$, $\mathtt{rect}$ for {\em rectangles}, $\mathtt{pi}$ and  $\mathtt{curve}$). This is followed by the size of the corresponding image -- the images are square, so it is enough to mention only one number. The name of the instance may also contain an additional keyword for better identification. Finally, the actual input instances are extended by the keywords $\mathtt{\_low}$, $\mathtt{\_med}$ and $\mathtt{\_high}$ that define how much noise was added.

The algorithm is run on these instances and after it finishes the minimization and finds a locally $\epsilon$-consistent point, we look on the returned values $h_i$, as defined in Section \ref{io_section}. If each cluster corresponding to a pixel has a unique maximum, we can reconstruct the solution by setting the corresponding maximum labels. On the other hand, if there are more maxima in a cluster, then we mark the object as undecided, which will be denoted by red pixels in the following section.

To find the optimal solutions to these instances, we calculated a continuous LP in form \eqref{dual} using Gurobi Optimizer version 7.5.2. We also compared the runtime of our implementation with the Gurobi Optimizer\footnote{The experiments were done on a computer with Intel Core i7 7500U CPU and 8 GB RAM.}. As described in \cite{gurobi}, the LP solver by default uses a concurrent solver which runs various optimization algorithms on more processor cores at the same time in parallel and returns the solution when the fastest algorithm terminates. Therefore, the runtime of Gurobi could be higher if it comitted to a single algorithm and used only one processor core at time.

It is necessary to mention that in order to be able to find integer solutions, we multiplied the values in the resulting $\mathbf{b}$ (resp. $\bm{g}$) vector by a large integer.

\section{Results}

\subsection{Lines Grammar}

In the Figure \ref{fig:results_line_grammar}, there are the results of the experiments with the {\em lines} grammar. In Figure \ref{fig:result4_lines}, there is the baseline image to which the noise was added, which created the input images in Figures \ref{fig:result1_lines}, \ref{fig:result2_lines} and \ref{fig:result3_lines}. After the minimization algorithm was applied, it found a solution, where each cluster contained exactly one subfunction that was alive (after the local consistency algorithm was run), which allowed to reconstruct the solution in Figure \ref{fig:result4_lines}. This solution is exactly the baseline image to which the noise was added and the result was the same in all the input images, the only difference is that with higher noise, more iterations were needed.

The same experiment was performed also with different image sizes, the input images in Figures \ref{fig:result7_lines}, \ref{fig:result8_lines} and \ref{fig:result9_lines} have the size 100 $\times$ 100 and used baseline image shown in Figure \ref{fig:result10_lines}. As in the previous case, the algorithm always found the solution that corresponds exactly to the baseline image.

In the previous cases, the image to which the noise was added was always in the grammar that we were searching, so we also tried to use a baseline image that was also used in \cite{4204160} which contains an inaccurately drawn square. This input image is in the original source called "diamond". To this image, we added all levels of noise, producing Figures \ref{fig:result5a_lines}, \ref{fig:result5b_lines} and \ref{fig:result5c_lines} and searched for the nearest image generated by the {\em lines} grammar. In the case of low and medium noise, the algorithm surprisingly found solutions in which all the maxima were unique. The result to the input with medium noise is in Figure \ref{fig:result6_lines}. The result corresponding to the low noise was not the same, but very similar -- the "cross" was a little bit thinner. In the case of high noise, no maximum of any cluster was unique, which prevented us from any reconstruction.

\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines200_low_noise_input.png}}
            \caption{Input image with low noise, size 200 $\times$ 200}   
            \label{fig:result1_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines50diamond_low_noise_input.png}}
            \caption{Input with baseline diamond image, low noise}
            \label{fig:result5a_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines100_low_noise_input.png}}
            \caption{Input image with low noise, size 100 $\times$ 100}   
            \label{fig:result7_lines}
        \end{subfigure} %% END LOW NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   %START MED NOISE
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines200_mid_noise_input.png}}
            \caption{Input image with med. noise, size 200 $\times$ 200}   
            \label{fig:result2_lines}
        \end{subfigure} 
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines50diamond_mid_noise_input.png}}
            \caption{Input with baseline diamond image, med. noise}
            \label{fig:result5b_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines100_mid_noise_input.png}}
            \caption{Input image with med. noise, size 100 $\times$ 100}   
            \label{fig:result8_lines}
        \end{subfigure} % END MID NOISE, START HIGH NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines200_high_noise_input.png}}
            \caption{Input image with high noise, size 200 $\times$ 200}    
            \label{fig:result3_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines50diamond_high_noise_input.png}}
            \caption{Input with baseline diamond image, high noise}
            \label{fig:result5c_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{inputlines100_high_noise_input.png}}
            \caption{Input image with high noise, size 100 $\times$ 100}    
            \label{fig:result9_lines}
        \end{subfigure} % END HIGH NOISE, START SOLUTIONS
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{resultlines200_low_noise_solved.png}}
            \caption{Both result and baseline image, size 200 $\times$ 200}
            \label{fig:result4_lines}
        \end{subfigure}
                \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{lines50diamond_variance_100_solved.png}}
            \caption{Result corresponding to the diamond input \ref{fig:result5b_lines}}
            \label{fig:result6_lines}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{resultlines100_low_noise_solved.png}}
            \caption{Both result and baseline image, size 100 $\times$ 100}
            \label{fig:result10_lines}
        \end{subfigure}
        \caption{Input images and solutions for the {\em lines} grammar.}
        \label{fig:results_line_grammar}
    \end{figure*}

Table \ref{table_numeric_results_all} shows the comparison between the optimal (minimal) values of the upper bound of various SMAF instances and the minimized values returned by the algorithm. The table also contains values that denote in which cases the height was truly optimal -- we express the difference between the optimal value $f(\mathbf{x}^*)$ and the found function value $f(\mathbf{x})$ as a value
\begin{equation}
R = \frac{f(\mathbf{x})-f(\mathbf{x}^*)}{f(\mathbf{x}^*)},
\end{equation}
which calculates the difference as a multiple of the optimal value. The table also contains the returned value of $\epsilon$ and the corresponding runtimes.

Our algorithm was significantly faster on the tested {\em lines} instances compared to the LP solver. On instances that are "obvious" to human perception, it produced the solution immediately. The other instances took more time.


\subsection{Rectangles Grammar}

Similar experiments to the previous ones were also performed with the {\em rectangles} grammar -- we created a baseline image that is generated by the grammar ($\mathtt{rect100sharp}$) and another one that is not ($\mathtt{rect100round}$) and then added noise with various variance to create the individual instances.

The generated images and the corresponding results are in Figure \ref{fig:results_rectangular_grammar}. The red pixels in the solutions mean that the cluster corresponding to the red pixel had multiple maxima alive, therefore it was not possible to determine the label.

The qualitative results are again in Table \ref{table_numeric_results_all} -- this time, there were fewer optima reached and the non-optimal values were also more distant from the optimum than in the case of {\em lines} grammar. Similarly as last time, the runtimes of our algorithm are significantly smaller than the runtimes of the LP solver.




\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100sharp_low_noise_input.png}}
            \caption{Input $\mathtt{rect100prec\_low}$}   
            \label{fig:result1_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100sharp_mid_noise_input.png}}
            \caption{Input $\mathtt{rect100prec\_med}$}
            \label{fig:result2_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100sharp_high_noise_input.png}}
            \caption{Input $\mathtt{rect100prec\_high}$}   
            \label{fig:result3_rect}
        \end{subfigure} %% END LOW NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   %START MED NOISE
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100sharp_low.png}}
            \caption{Result of $\mathtt{rect100sharp\_low}$}   
            \label{fig:result4_rect}
        \end{subfigure} 
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100sharp_mid.png}}
            \caption{Result of $\mathtt{rect100sharp\_med}$}
            \label{fig:result5_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100sharp_high.png}}
            \caption{Result of $\mathtt{rect100sharp\_high}$}   
            \label{fig:result6_rect}
        \end{subfigure} % END MID NOISE, START HIGH NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100round_low_noise_input.png}}
            \caption{Input $\mathtt{rect100round\_low}$}    
            \label{fig:result7_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100round_mid_noise_input.png}}
            \caption{Input $\mathtt{rect100round\_med}$}
            \label{fig:result8_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{rect100round_high_noise_input.png}}
            \caption{Input $\mathtt{rect100round\_high}$}    
            \label{fig:result9_rect}
        \end{subfigure} % END HIGH NOISE, START SOLUTIONS
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100round_low.png}}
            \caption{Result of $\mathtt{rect100round\_low}$}
            \label{fig:result10_rect}
        \end{subfigure}
                \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100round_mid.png}}
            \caption{Result of $\mathtt{rect100round\_med}$}
            \label{fig:result11_rect}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100round_high.png}}
            \caption{Result of $\mathtt{rect100round\_high}$}
            \label{fig:result12_rect}
        \end{subfigure}
        \caption{Input images and solutions for the {\em rectangles} grammar.}
        \label{fig:results_rectangular_grammar}
    \end{figure*}

\subsection{Pi Grammar}\label{pi_grammar_results}

In case of the {\em pi} grammar, an image which is not generated by the grammar (called $\mathtt{pi50rough}$) and also an image that originally is generated by the grammar (called $\mathtt{pi50prec}$) were used. The input and output images are shown in Figure \ref{fig:results_russian_p_grammar}.

In the cases of low and medium noise, there are few undecided points, which are interestingly located only in the upper part of the image. The reason for this can be easily discovered after finding out which labels correspond to the clusters' maxima. In case of Figures \ref{fig:result4_rus} and \ref{fig:result10_rus}, the lowest undecided line has two maxima -- it could be an empty space P or the lower boundary LoB of a pi sign. The upper lines have three maxima -- it could be empty space P (if the pixel below was empty), lower boundary LoB (if the pixel below was empty) or interior point I (if the pixel below was a lower boundary or also an interior point). So, it can be basically viewed as that it is known that the red points in these images should be white, but it is not known if they should be empty or if there could be a large pi sign whose black pixels are out of the picture and therefore the upper part of the image could be inside such sign. This does not happen in the lower part of the image because the upper boundary of the pi sign would be visible there.

On the other hand, in the case of the images with medium noise, i.e. the results that are in Figures \ref{fig:result5_rus} and \ref{fig:result11_rus}, few pixels in the uppermost row are so dark due to the noise that they should be black and maxima of their clusters correspond to the LeLo and RiLo labels, i.e. the end of either the left or the right vertical line of the pi symbol. The pixels between them are either empty (label P) or the lower boundary (LoB) of a pi symbol -- this would be based on the choice of the left/right direction of the other labels. So, we found multiple labelings with the highest quality.

In the results corresponding to the input with high noise, the undecided pixels are not so structured, likely because the input does not resemble an image from the pi grammar at all.

The numerical results on these instances are listed in Table \ref{table_numeric_results_all}, where it can be seen that the runtime of our algorithm was again significantly lower than the runtime of the LP solver and in 4 of the 6 cases, we reached the optimum.


\begin{figure*}
        \centering
        
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50rough_low.png}}
            \caption{Input $\mathtt{pi50rough}$ with low noise}   
            \label{fig:result1_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50rough_mid.png}}
            \caption{Input $\mathtt{pi50rough}$ with medium noise}
            \label{fig:result2_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50rough_high.png}}
            \caption{Input $\mathtt{pi50rough}$ with high noise}   
            \label{fig:result3_rus}
        \end{subfigure} %% END LOW NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   %START MED NOISE
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50rough_low.png}}
            \caption{Result of $\mathtt{pi50rough}$ with low noise}   
            \label{fig:result4_rus}
        \end{subfigure} 
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50rough_mid.png}}
            \caption{Result of $\mathtt{pi50rough}$ with medium noise}
            \label{fig:result5_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50rough_high.png}}
            \caption{Result of $\mathtt{pi50rough}$ with high noise}   
            \label{fig:result6_rus}
        \end{subfigure} % END MID NOISE, START HIGH NOISE
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50prec_low.png}}
            \caption{Input $\mathtt{pi50prec}$ with low noise}    
            \label{fig:result7_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50prec_mid.png}}
            \caption{Input $\mathtt{pi50prec}$ with medium noise}
            \label{fig:result8_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{russianp50prec_high.png}}
            \caption{Input $\mathtt{pi50prec}$ with high noise}    
            \label{fig:result9_rus}
        \end{subfigure} % END HIGH NOISE, START SOLUTIONS
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50prec_low.png}}
            \caption{Result of $\mathtt{pi50prec}$ with low noise}
            \label{fig:result10_rus}
        \end{subfigure}
                \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50prec_mid.png}}
            \caption{Result of $\mathtt{pi50prec}$ with medium noise}
            \label{fig:result11_rus}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_russianp50prec_high.png}}
            \caption{Result of $\mathtt{pi50prec}$ with high noise}
            \label{fig:result12_rus}
        \end{subfigure}
        \caption{Input images and solutions for the {\em pi} grammar.}
        \label{fig:results_russian_p_grammar}
    \end{figure*}
    
\subsection{Curve Grammar}

In the {\em curve} grammar, we generated the largest instance, which was made up from an image with size 500 $\times$ 500 that is originally generated by the grammar. Again, we added various amounts of noise to the image and tried to minimize the height of the corresponding binary max-sum problem. The input images and the results are shown in Figure \ref{fig_results_curve}. 

Similarly as in the previous cases, the comparison of runtimes along with the optimality of results is shown in Table \ref{table_numeric_results_all}. This time, Gurobi could not run the concurrent solver because it required too much memory. Neither dual simplex method nor the barrier method could be used even individually because of the same reason. That is why the runtimes in this case are calculated by the primal simplex method. In case of $\mathtt{curve500\_high}$, the calculations ran more than 2 weeks before out of memory error occurred. That is why the optimum of this instance was not calculated.

On the other hand, our algorithm terminated with a much slower runtime on these instances and reached the optimal value in the case with medium and low noise. By Theorem \ref{theorem_optimality_1}, we know that on the instance with high noise, suboptimal value was reached even without knowing the optimal value of the problem.

The runtimes in Table \ref{table_numeric_results_all} are measured only for the iterative minimization procedure, we did not measure the runtime of loading the input file or saving the result. Similarly for the LP solver, we did not measure the time required for the set-up of the LP, but only the minimization process itself.

\begin{figure}
        \centering
        
        \begin{subfigure}[b]{0.42\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{input_curve500_low.png}}
            \caption{Input $\mathtt{curve500}$ with low noise}   
            \label{fig:result1_curve}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.42\textwidth}   %START MED NOISE
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_curve500_low.png}}
            \caption{Result of $\mathtt{curve500}$ with low noise}   
            \label{fig:result4_curve}
        \end{subfigure}
        
        
        \begin{subfigure}[b]{0.42\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{input_curve500_mid.png}}
            \caption{Input $\mathtt{curve500}$ with medium noise}
            \label{fig:result2_curve}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.42\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_curve500_med_fixed.png}}
            \caption{Result of $\mathtt{curve500}$ with medium noise}
            \label{fig:result5_curve}
        \end{subfigure}
        
        
        \begin{subfigure}[b]{0.42\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{input_curve500_high.png}}
            \caption{Input $\mathtt{curve500}$ with high noise}   
            \label{fig:result3_curve}
        \end{subfigure} %% END LOW NOISE        
        \quad
        \begin{subfigure}[b]{0.42\textwidth}  
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_curve500_high_fixed.png}}
            \caption{Result of $\mathtt{curve500}$ with high noise}   
            \label{fig:result6_curve}
        \end{subfigure}
        
        \caption{Input images and solutions for the {\em curve} grammar.}        \label{fig_results_curve}
    \end{figure}
    
   
\begin{landscape}
\begin{table}
\centering
%\hspace{-0.7cm}
 %\vspace{-1cm}
    \begin{tabular}{l|D{.}{.}{3}|r|r|r|c|c}
    \textbf{Instance of SMAF} & \textbf{Optimum} & \textbf{Minimized to} & \textbf{LP runtime} & \textbf{Our runtime}& $\mathbf{\epsilon}$ & $R$ \\    \hline
 \texttt{lines200\_high} &14194617352192 &14194617352192 & 14815.0 s &220.4 s & 0& 0 \\
 \texttt{lines200\_med} & 18093229211648&18093229211648 &  1983.0 s& 1.1 s & 0&0  \\
 \texttt{lines200\_low} &20396852117504 &20396852117504 &  1539.5 s & $<$ 1 s  & 0 & 0  \\ \hline
 \texttt{lines100\_high} &3553387085824 & 3553387085824 & 925.7 s & 24.1 s & 0& 0\\
 \texttt{lines100\_med} & 4519050084352 & 4519050084352&  826.4 s & $<$ 1 s & 0&  0 \\
 \texttt{lines100\_low} & 5098038099968 &5098038099968 &   849.8 s &$<$ 1 s  & 0 & 0\\  \hline
 \texttt{lines50diamond\_high} & 918061842432& 918061842662  &  44.6 s & $<$ 1 s & 3 & $2.5 \cdot 10^{-10}$ \\ 
 \texttt{lines50diamond\_med} & 939794628608& 939794628608 &  23.6 s& 1.9 s & 0 & 0 \\ 
 \texttt{lines50diamond\_low} & 1025186463744& 1025186463744 &   22.5 s& 1.5 s & 0 & 0 \\ \hline
 \texttt{rect100sharp\_high} &3862483641786.583 &3862948035702  & 485.6 s & 5.4 s & 82 & $1.2 \cdot 10^{-4}$ \\
 \texttt{rect100sharp\_med} & 4522357293056 & 4522357293056& 6585.6 s & $<$ 1 s& 0&0 \\
 \texttt{rect100sharp\_low} &5098293952512 & 5098293952512&  4338.2 s &  $<$ 1 s& 0 & 0 \\ \hline
 \texttt{rect100round\_high} &3858045642353.665 & 3858719475728 &  5427.7 s & 17.3 s & 2453& $1.7 \cdot 10^{-4}$ \\
 \texttt{rect100round\_med} & 4469794039592.592 & 4470672873012 &   7427.2 s & 61.4 s& 9191&  $2.0 \cdot 10^{-4}$\\
 \texttt{rect100round\_low} & 5019918069332.757 & 5020900572035 &  5314.2 s &72.5 s & 40231 & $2.0 \cdot 10^{-4}$\\ \hline
 \texttt{pi50rough\_high} & 1001347577719.467 &1001419353514  &  170.6 s &$< 1$ s & 3 & $7.2 \cdot 10^{-5}$\\
 \texttt{pi50rough\_med} & 1113698861056 & 1113698861056&  168.7 s & $< 1$ s& 0&0 \\
 \texttt{pi50rough\_low} &1233886642176 & 1233886642176&  188.4 s  & $< 1$ s& 0 & 0 \\ \hline
 \texttt{pi50prec\_high} &1006988427264 &  1007074368367
 &  105.6 s & $< 1$ s & 3& $8.5 \cdot 10^{-5}$ \\
 \texttt{pi50prec\_med} & 1139051331584 & 1139051331584&    183.5 s & $< 1$ s& 0&  0  \\
 \texttt{pi50prec\_low} & 1276033105920 &1276033105920 &   180.5 s  &$< 1$ s & 0 & 0 \\ \hline
  \texttt{curve500\_high} & \text{n/a} & 104703708253010 &  $> 1.5 \cdot 10^6$ s  & 220.9 s &  32& $> 0$  \\
 \texttt{curve500\_med} & 112950824140800 & 112950824140800  & 32524.1 s & 26.3 s & 0 &0 \\
 \texttt{curve500\_low} & 127430662553600 & 127430662553600&   39570.3 s & 4.1 s & 0& 0
    \end{tabular}
       \caption{Overview of all the numerical results and runtimes on all instances.}
    \label{table_numeric_results_all}
\end{table}
\end{landscape}

Except for a few instances, we are able to determine the optimality of the solutions just by applying Theorems \ref{theorem_optimality_1} and \ref{theorem_optimality_2} and we would not even have to know the optimal values from the LP. In the instances $\mathtt{pi50rough}$ and $\mathtt{pi50prec}$ with medium and low noise, no previously shown theorem can decide the optimality of the result, but it could be deduced from the reasoning in Section \ref{pi_grammar_results} because we know that these instances of SMAF correspond to the upper bound of a binary max-sum problem. Therefore, the only case that would remain unresolved if we did not know the true optimum would be the $\mathtt{curve500\_med}$ instance.

\subsection{Small Changes}

We have also experimented with changing the values in the $\mathbf{b}$ vector to re-optimize the problem for a slightly different setting.

\begin{figure}[h]
        \centering
	\begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_pi50prec_low_changed.png}}
            \caption{Result of modified $\mathtt{pi50prec\_low}$ instance, the red lines disappear}   
            \label{fig:result0_changes}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_pi50prec_mid_changed.png}}
            \caption{Result of modified $\mathtt{pi50prec\_med}$ instance, the red line disappeared}   
            \label{fig:result1_changes}
        \end{subfigure}
        \quad
        
        
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100prec_high_changed_by500M.png}}
            \caption{Result of modified $\mathtt{rect100prec\_high}$ instance with $\omega = 1$}
            \label{fig:result2_changes}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100prec_high_changed_by5G.png}}
            \caption{Result of modified $\mathtt{rect100prec\_high}$ instance with $\omega = 10$}   
            \label{fig:result3_changes}
        \end{subfigure}
        
        
        \quad
        \begin{subfigure}[b]{0.3\textwidth}   
            \centering 
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100round_high_changed_by500M.png}}
            \caption{Result of modified $\mathtt{rect100round\_high}$ instance with $\omega = 1$}
            \label{fig:result4_changes}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.3\textwidth}
            \centering
            \fcolorbox{blue}{blue}{\includegraphics[width=\textwidth]{result_rect100round_high_changed_by5G.png}}
            \caption{Result of modified $\mathtt{rect100round\_high}$ instance with $\omega = 10$}   
            \label{fig:result5_changes}
        \end{subfigure}
\label{fig_results_changes}
        \caption{Results corresponding to modified instances.}
    \end{figure}
    
\subsubsection{Pi Grammar Instances}

In this case, the possibility to do small changes of the problem become useful -- we can help the solver to decide which solution to take. For example, in the case of $\mathtt{pi50prec\_low}$ or $\mathtt{pi50rough\_low}$, we can change e.g. the value $b_{i,j}$ corresponding to the empty label of the pixel in the upper left corner and increase it by 1. After that, the algorithm does not need to change the current value of $\mathbf{x}$ in this case because it still is locally consistent but the ambiguous labels disappear because in the upper left corner, there is only one maximum -- the empty label. Therefore the I or LoB labels become inconsistent and are killed, which means that the previously red areas in Figures \ref{fig:result4_rus} and \ref{fig:result10_rus} now become white because the only remaining subfunctions that are alive correspond to empty label. The result corresponding to the modified instance $\mathtt{pi50prec\_low}$ is in Figure \ref{fig:result0_changes}.

The same thing can be done also for the instances with medium noise, which would result in an image exactly like Figures \ref{fig:result5_rus}, resp. \ref{fig:result11_rus}, but in the uppermost row, there would be few separated black pixels. The result corresponding to the modified instance $\mathtt{pi50prec\_med}$ is in Figure \ref{fig:result1_changes}.

\subsubsection{Rectangles Grammar Instances}

In the results of the {\em rectangles} grammar instances with high noise, there was a lot of small rectangles, i.e. in Figures \ref{fig:result6_rect} and \ref{fig:result12_rect}. In order to try to improve the result, we tried to introduce a regularizer on the amount of rectangles.

The regularizer is the decrease of the $b_{i,j}$ values corresponding to the corner labels of the rectangles by a value $\omega$. In this way, the optimization criterion \eqref{closest_image_measure} becomes
\begin{equation}
4 \, \omega \cdot \text{rectangles}(\mathbf{O}^*) + \sum_{i \in [h]}\sum_{j \in [w]} \abs{o^*_{i,j} - o_{i,j}},
\end{equation}
where $\text{rectangles}(\mathbf{O}^*)$ is the amount of rectangles in the image $\mathbf{O}^*$.

We tried to set $\omega  = 1$ or $\omega  = 10$ and decrease the corresponding $b_{i,j}$ values by this value\footnote{Recall that the values $b_{i,j}$ were multiplied by a large integer, these $\omega$ values are therefore also actually multiplied by the same value.}.

In the case of $\mathtt{rect100prec\_high}$, the smaller $\omega$ resulted in Figure \ref{fig:result2_changes}, which contains fewer rectangles than the original result without regularization. The larger value of $\omega$ returned the result in Figure \ref{fig:result3_changes}, which almost exactly corresponds to the baseline image, except that the upper left rectangle is narrower by 1 pixel from each side and the left long rectangle is wider by 2 pixels on the left side.

The same worked also for the instance $\mathtt{rect100round\_high}$, whose corresponding result for $\omega = 1$ is in Figure \ref{fig:result4_changes}, which again contains fewer rectangles. Figure \ref{fig:result5_changes} shows the result for $\omega = 10$ -- notice that the result contains four rectangles in the places that can be recognized e.g. in Figure \ref{fig:result7_rect}.

The recalculation of the problem from the previously locally consistent point is beneficial in the means of runtime. In all these cases, the runtime of the algorithm after the small change was performed was smaller than if the algorithm was completely restarted. This could be also beneficial in cases where the regularization constant is searched and results for multiple regularizations are compared.

\section{Further Analysis}

Here, we first analyse the dependence of the number of iterations and the final function value on the initial value of $\epsilon$. Then, we look into one iteration and its progress during a single run.

\subsection{Dependence on Initial Value of $\epsilon$}

As we said above, we introduce $\epsilon$-consistency in order to make the calculations faster. Now, we will show how it is sped up based on the initial value of $\epsilon$. We analysed the speed-up by the number of iterations required for the algorithm to terminate.

We observed that the number of iterations does not always increase when initial $\epsilon$ is lowered and that the instances can be roughly divided into two groups. In the first one, the number of iterations is initially almost constant with increasing value of initial $\epsilon$ but after a region of values of the initializer is crossed, the number of iterations decreases rapidly. Examples of such instances are for example $\mathtt{rect100sharp\_med}$ or $\mathtt{lines200\_med}$, for whose the required amount of iterations for various initial values of $\epsilon$ is shown in Figures \ref{fig:epsilon_iterations0} and \ref{fig:epsilon_iterations1}. For these instances, the reached function value was the same for all initialization values or at least very similar. This case usually happened in the "easier" instances.

\begin{figure}[h]
        \centering
	\begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_iterations_rect100sharphigh_zoomed.eps}
            \caption{Number of iterations for instance $\mathtt{rect100sharp\_med}$}   
            \label{fig:epsilon_iterations0}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_iterations_lines200mid_zoomed.eps}
            \caption{Number of iterations for instance $\mathtt{lines200\_med}$}   
            \label{fig:epsilon_iterations1}
        \end{subfigure}
        	\begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_iterations_rect100roundhigh_zoomed.eps}
            \caption{Number of iterations for instance $\mathtt{rect100round\_high}$}   
            \label{fig:epsilon_iterations2_0}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_iterations_lines100high_zoomed.eps}
            \caption{Number of iterations for instance $\mathtt{lines100\_high}$}   
            \label{fig:epsilon_iterations2_1}
        \end{subfigure}
        \begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_function_value_rect100roundhigh_zoomed.eps}
            \caption{Reached function value for instance $\mathtt{rect100round\_high}$}   
            \label{fig:epsilon_iterations2_2}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.45\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_function_value_lines100high_zoomed.eps}
            \caption{Reached function value for instance $\mathtt{lines100\_high}$}   
            \label{fig:epsilon_iterations2_3}
        \end{subfigure}
\label{fig_epsilon_initiz}
        \caption{Comparison of results for various initialization values of $\epsilon$.}
    \end{figure}

In the other group, which basically contains the "harder" instances, the amount of iterations first increases with increasing initial $\epsilon$ but then starts to decrease. Examples of such progress are shown in Figures \ref{fig:epsilon_iterations2_0} and \ref{fig:epsilon_iterations2_1} that corresponds to instances $\mathtt{rect100round\_high}$ and $\mathtt{lines100\_high}$. It might seem that having small initial value of $\epsilon$ could be also beneficial in this cases but the contrary holds because the low amount of iterations for small initial values is caused by worse results of the algorithm. It means that even though the algorithm terminates faster, it ends with a larger function value. Figures \ref{fig:epsilon_iterations2_2} and \ref{fig:epsilon_iterations2_3} show the function value with that the algorithm terminates on these instances based on the initial value of $\epsilon$. The function value for small initializers is higher than for the higher ones.


In all the figures, the maximum value of initial $\epsilon$ is 
\begin{equation}
\bar{\epsilon} = \max_{i,j} b_{i,j} - \min_{i,j} b_{i,j},
\end{equation}
which is the initialization proposed in Algorithm \ref{decreasing_epsilon_integer_alg}, where it was also justified why larger values would not change the outcome. From the previous experimental observations, it follows that this initializer lowers the required amount of iteration and also provides lower function values when compared to the other ones.

\subsection{Typical Progress of Objective Value}

Here, we look into a single run of the algorithm -- we will show the results for instances $\mathtt{lines200\_high}$ and $\mathtt{rect100sharp\_med}$, for the other instances, the results would be similar\footnote{Except for those that are locally consistent already at $\mathbf{x} = 0$. These finish immediately without any iterations and are therefore uninteresting.}.

First of all, we show how the current function value decreases during the minimization process. This is shown in Figures \ref{figure_iteration_convergence_epsilon_trimmed} and \ref{figure_iteration_convergence_epsilon_2}. These figures contain multiple plots, each for a different initial $\epsilon$ value. Since we already know from the previous section that the convergence is basically the fastest with $\epsilon = \bar{\epsilon}$, the result of comparison of the initial $\epsilon$ values is not surprising. The shapes of the curves in these figures could be denoted as "typical" results of optimization methods in the sense that it initially decreases steeply but as the value approaches the optimum, the decrease slows down.

The decrease of the function in a single iteration is connected to the current value of $\epsilon$. In Figures \ref{figure_lines_stepsize_epsilon} and \ref{figure_lines_stepsize_epsilon_2}, we can see by what value the function decreased in each iteration and what was the current value of $\epsilon$ at that time. This is the run, where $\epsilon$ was initialized as $\bar{\epsilon}$. From both figures, we can see that larger values of $\epsilon$ correspond to a larger decrease in the objective function value. These figures also explain why the convergence with significantly lower $\epsilon$ is slower -- because the initial value is small, it is not possible to make the large steps in the beginning as with the larger initializations.

Note that in the case of Figure \ref{figure_lines_stepsize_epsilon}, the final value was reached already with $\epsilon \approx 2 \cdot 10^6$ and the value of $\epsilon$ was then always lowered without changing $\mathbf{x}$. This is why the plot is trimmed for clarity and we do not see how $\epsilon$ is lowered to zero. 


\begin{figure}[h]
        \centering
	\begin{subfigure}[b]{0.8\textwidth}
            \centering
            \includegraphics[width=\textwidth]{initial_epsilon_lines_trimmed.eps}
            \caption{Plot showing the current value of the function with respect to the current iteration and the initial value of $\epsilon$. Notice that the horizontal axis is trimmed and the run corresponding to $\epsilon = 10^6$ terminated after approximately $2.8 \cdot 10^5$ iterations.}   
            \label{figure_iteration_convergence_epsilon_trimmed}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.8\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_lines_stepsize_epsilon_fixed.eps}
            \caption{Plot showing the decrease of the function value and the current value of $\epsilon$ during a single run of the algorithm, initialized with $\bar{\epsilon}$.}   
            \label{figure_lines_stepsize_epsilon}
        \end{subfigure}
        \caption{Progress of minimization, instance $\mathtt{lines200\_high}$.}
    \end{figure}
    
\begin{figure}[h]
        \centering
	\begin{subfigure}[b]{0.8\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_decrease_in_function_value.eps}
            \caption{Plot showing the current value of the function with respect to the current iteration and the initial value of $\epsilon$.}   
            \label{figure_iteration_convergence_epsilon_2}
        \end{subfigure}
        \quad
        \begin{subfigure}[b]{0.8\textwidth}
            \centering
            \includegraphics[width=\textwidth]{plot_lines_stepsize_epsilon_second_image.eps}
            \caption{Plot showing the decrease of the function value and the current value of $\epsilon$ during a single run of the algorithm, initialized with $\bar{\epsilon}$.}   
            \label{figure_lines_stepsize_epsilon_2}
        \end{subfigure}
        \caption{Progress of minimization, instance $\mathtt{rect100sharp\_med}$.}
    \end{figure}