Image Component Library (ICL)
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines
Classes | Public Member Functions | Static Public Attributes | Private Member Functions | Private Attributes
icl::math::LLM Class Reference

Local Linear Map implementation (LLM) More...

#include <LLM.h>

Inheritance diagram for icl::math::LLM:
icl::utils::Configurable

List of all members.

Classes

struct  Kernel
 Internally used Kernel structure. More...

Public Member Functions

 LLM (unsigned int inputDim, unsigned int outputDim)
 Creates a new llm with given input and output dimensions.
 LLM (unsigned int inputDim, unsigned int outputDim, unsigned int numCenters, const std::vector< utils::Range< icl32f > > &ranges, const std::vector< float > &var=std::vector< float >(1, 1))
void init (unsigned int numCenters, const std::vector< utils::Range< icl32f > > &ranges, const std::vector< float > &var=std::vector< float >(1, 1))
 initializes the LLM prototypes with given kernel count
void init (const std::vector< float * > &centers, const std::vector< float > &var=std::vector< float >(1, 1))
 initializes the LLM prototypes with given set of kernels
const float * apply (const float *x)
 applies the LLM and returns current NET output vector
void train (const float *x, const float *y, int trainflags=TRAIN_ALL)
 applies an update step to the net with given net input x and destination output y
void trainCenters (const float *x)
 trains the input weights w_in
void trainSigmas (const float *x)
 trains the input variances sigma
void trainOutputs (const float *x, const float *y)
 trains the output weights w_out
void trainMatrices (const float *x, const float *y)
 trains the output slope matrices A
const float * getErrorVec (const float *x, const float *y)
 returns the current error vector with respect to a given destination value y
void setEpsilonIn (float val)
 sets up learning rate for input weights to a new value (about 0..1)
void setEpsilonOut (float val)
 sets up learning rate for output weights to a new value (about 0..1)
void setEpsilonA (float val)
 sets up learning rate for slope matrices to a new value (about 0..1)
void setEpsilonSigma (float val)
 sets up learning rate for sigmas to a new value (about 0..1)
void showKernels () const
 Shows all current kernels to std::out.
unsigned int numKernels () const
 returns the current internal kernel count
const Kerneloperator[] (unsigned int i) const
 returns a specifice kernel at given index (const)
Kerneloperator[] (unsigned int i)
 returns a specifice kernel (unconst version)
bool isSoftMaxUsed () const
 returns whether the softmax function for calculation for g_i[x] is used
void setSoftMaxEnabled (bool enabled)
 sets whether the softmax function for calculation for g_i[x] is used

Static Public Attributes

static const int TRAIN_CENTERS = 1
static const int TRAIN_SIGMAS = 2
static const int TRAIN_OUTPUTS = 4
static const int TRAIN_MATRICES = 8
static const int TRAIN_ALL = TRAIN_CENTERS | TRAIN_SIGMAS | TRAIN_OUTPUTS | TRAIN_MATRICES

Private Member Functions

void init_private (unsigned int inputDim, unsigned int outputDim)
const float * updateGs (const float *x)
 possible usefull utility function (internally used)
void trainCentersIntern (const float *x, const float *g)
 internal training function for the input weights/prototypes
void trainSigmasIntern (const float *x, const float *g)
 interal training function for the sigmas
void trainOutputsIntern (const float *x, const float *y, const float *g, const float *dy, bool useDeltaWin)
 internal training function for the output weights
void trainMatricesIntern (const float *x, const float *y, const float *g, const float *dy)
 internal training function for the slope matrices A
const float * applyIntern (const float *x, const float *g)
 internal apply function
const float * getErrorVecIntern (const float *y, const float *ynet)
 internal function to return the current error vector

Private Attributes

unsigned int m_inputDim
 input dimension
unsigned int m_outputDim
 output dimension
std::vector< Kernelm_kernels
 internal storage for the kernels
std::vector< float > m_outBuf
 internal output value buffer
std::vector< float > m_gBuf
 internal buffer for the g_i[x]
std::vector< float > m_errorBuf
 internal buffer for the current error vector

Detailed Description

Local Linear Map implementation (LLM)

The LLM-network (Local Linear Maps) is a very powerful regression network that uses a superposition of linearily weighted gaussians as regression model:

Net Function y=f(x)

\[ \vec{y}^{net}(x) = \sum\limits_{i=1}^N ( \vec{w}_i^{out}+A_i ( \vec{x}-\vec{w}_i^{in} ) ) \cdot g_i(\vec{x}) \]

where each kernel i is defined by a kernel function $g_i(\vec{x})$ that is weighted linearily using $ ( \vec{w}_i^{out}+A_i ( \vec{x}-\vec{w}_i^{in} ) ) $. $\vec{w}_i^{in}$ is the location in the input space, where kernel i is centered, and $\vec{w}_i^{out}$ resp. $A_i$ are parameters of a linear weighting function.

Kernel functions

Here, normalized Gaussians are always used a kernel functions:

\[ g_i(\vec{x}) = \frac{\exp(-\frac{\|\vec{x}-\vec{w}_i^{in}\|^2}{2\sigma_i^2})} {\sum\limits_{j=1}^N \exp(-\frac{\|\vec{x}-\vec{w}_j^{in}\|^2}{2\sigma_j^2})} \]

Training of input weights and variances(unsuperwised)

The kernel centers (also refered as data prototypes) $\vec{w}_i^{in}$ can be initialized randomly e.g. using a uniform distribution in input data space or by a set of given prototypes. The prototypes can be adapted while the network training process using means of vector quantisation (e.g. KMeans) externally or by using the following internally implemented update rule:

\[ \Delta \vec{w}_i^{in} = \epsilon_{in} \cdot ( \vec{x} - \vec{w}_i^{in} ) \cdot g_i(\vec{x}) \]

and

\[ \Delta \sigma^2 = \epsilon_{\sigma} \cdot ( \vec{x} - \vec{w}_i^{in} )^2 - \sigma^2 \cdot g_i(\vec{x}) \]

Note: For some initial tries, it turned out, that updating the kernel radii is rather unstable.

Training of output weights and matrices (superwised)

Training rules for output weights $\vec{w}_i^{out}$ and matrices $A_i$ are obtained by simple gradient descent approach (given input tuple $(\vec{x}^{\alpha},\vec{y}^{\alpha})$):

\[ \Delta \vec{w}_i^{out} = \epsilon_{out} \cdot g_i(\vec{x}^{\alpha}) \cdot (\vec{y}^{\alpha} - \vec{y}^{net}(\vec{x}^{\alpha}) ) + A_i \Delta \vec{w}_i^{in} \]

and

\[ \Delta A_i = \epsilon_A \cdot g_i(\vec{x}^{\alpha}) \cdot (\vec{y}^{\alpha} - \vec{y}^{net}(\vec{x}^{\alpha}) ) \cdot \frac{(\vec{x}^{\alpha} - \vec{w}_i^{in} )^{\tau}}{\|\vec{x}^{\alpha} - \vec{w}_i^{in}\|^2} \]

Hacks

TODO


Constructor & Destructor Documentation

icl::math::LLM::LLM ( unsigned int  inputDim,
unsigned int  outputDim 
)

Creates a new llm with given input and output dimensions.

icl::math::LLM::LLM ( unsigned int  inputDim,
unsigned int  outputDim,
unsigned int  numCenters,
const std::vector< utils::Range< icl32f > > &  ranges,
const std::vector< float > &  var = std::vector< float >(1, 1) 
)

Member Function Documentation

const float* icl::math::LLM::apply ( const float *  x)

applies the LLM and returns current NET output vector

the ownership of the returnd float * is not passed to the caller. The output remains valid until apply is called with another x-value

const float* icl::math::LLM::applyIntern ( const float *  x,
const float *  g 
) [private]

internal apply function

const float* icl::math::LLM::getErrorVec ( const float *  x,
const float *  y 
)

returns the current error vector with respect to a given destination value y

const float* icl::math::LLM::getErrorVecIntern ( const float *  y,
const float *  ynet 
) [private]

internal function to return the current error vector

void icl::math::LLM::init ( unsigned int  numCenters,
const std::vector< utils::Range< icl32f > > &  ranges,
const std::vector< float > &  var = std::vector< float >(1, 1) 
)

initializes the LLM prototypes with given kernel count

Internally creates numCenters kernels for this LLM. Each kernel is initialized as follows:

  • input weight w_in: created with respect to ranges (w_in[i] becomes a value drawn from a uniform distibution of range ranges[i])
  • output weight w_out: initialized to 0-vector
  • Slope Matrix A: initialized to 0-matrix
  • variance sigma: initialized to given variance vector var

(internally this init function calls the other init function)

Parameters:
numCentersnew prototype count
rangesranges for uniform distributed prototypes (ranges.size must be equal to the input dimension of the LLM)
varvariance for the new created Gaussians (var.size must be equal to the input dimension of the LLM)
void icl::math::LLM::init ( const std::vector< float * > &  centers,
const std::vector< float > &  var = std::vector< float >(1, 1) 
)

initializes the LLM prototypes with given set of kernels

Internall all centers.size() vectors are used to create that many prototypes. The given vectors are deeply copied and not released internally. Initialization is performend as described in the above function.

void icl::math::LLM::init_private ( unsigned int  inputDim,
unsigned int  outputDim 
) [private]
bool icl::math::LLM::isSoftMaxUsed ( ) const [inline]

returns whether the softmax function for calculation for g_i[x] is used

unsigned int icl::math::LLM::numKernels ( ) const [inline]

returns the current internal kernel count

const Kernel& icl::math::LLM::operator[] ( unsigned int  i) const [inline]

returns a specifice kernel at given index (const)

Kernel& icl::math::LLM::operator[] ( unsigned int  i) [inline]

returns a specifice kernel (unconst version)

please note: it is not safe to change the obtained Kernel's value by hand unless you really know what you are doing. For setting the kernel weights manually, it is strongly recommended to use the Kernel::set-method, which does only allow to change the weight values

void icl::math::LLM::setEpsilonA ( float  val) [inline]

sets up learning rate for slope matrices to a new value (about 0..1)

void icl::math::LLM::setEpsilonIn ( float  val) [inline]

sets up learning rate for input weights to a new value (about 0..1)

void icl::math::LLM::setEpsilonOut ( float  val) [inline]

sets up learning rate for output weights to a new value (about 0..1)

void icl::math::LLM::setEpsilonSigma ( float  val) [inline]

sets up learning rate for sigmas to a new value (about 0..1)

Note Update of the sigmas does not run very good!

void icl::math::LLM::setSoftMaxEnabled ( bool  enabled) [inline]

sets whether the softmax function for calculation for g_i[x] is used

Shows all current kernels to std::out.

void icl::math::LLM::train ( const float *  x,
const float *  y,
int  trainflags = TRAIN_ALL 
)

applies an update step to the net with given net input x and destination output y

The trainingflags parameter can be used to specify special parameter updates, a list of training targets can be creates using bit-or: e.g.: TRAIN_CENTERS|TRAIN_SIGMAS

void icl::math::LLM::trainCenters ( const float *  x)

trains the input weights w_in

void icl::math::LLM::trainCentersIntern ( const float *  x,
const float *  g 
) [private]

internal training function for the input weights/prototypes

void icl::math::LLM::trainMatrices ( const float *  x,
const float *  y 
)

trains the output slope matrices A

void icl::math::LLM::trainMatricesIntern ( const float *  x,
const float *  y,
const float *  g,
const float *  dy 
) [private]

internal training function for the slope matrices A

void icl::math::LLM::trainOutputs ( const float *  x,
const float *  y 
)

trains the output weights w_out

void icl::math::LLM::trainOutputsIntern ( const float *  x,
const float *  y,
const float *  g,
const float *  dy,
bool  useDeltaWin 
) [private]

internal training function for the output weights

void icl::math::LLM::trainSigmas ( const float *  x)

trains the input variances sigma

void icl::math::LLM::trainSigmasIntern ( const float *  x,
const float *  g 
) [private]

interal training function for the sigmas

const float* icl::math::LLM::updateGs ( const float *  x) [private]

possible usefull utility function (internally used)

updates an internal variable vector containing provisional results for all g_i[x]. This internally speeds up performance (please dont call this function!, propably it can be made private)


Member Data Documentation

std::vector<float> icl::math::LLM::m_errorBuf [private]

internal buffer for the current error vector

std::vector<float> icl::math::LLM::m_gBuf [private]

internal buffer for the g_i[x]

unsigned int icl::math::LLM::m_inputDim [private]

input dimension

std::vector<Kernel> icl::math::LLM::m_kernels [private]

internal storage for the kernels

std::vector<float> icl::math::LLM::m_outBuf [private]

internal output value buffer

unsigned int icl::math::LLM::m_outputDim [private]

output dimension

training flag for updating all

const int icl::math::LLM::TRAIN_CENTERS = 1 [static]

training flag for updating input weights/prototype vectors

const int icl::math::LLM::TRAIN_MATRICES = 8 [static]

training flag for updating output matrices

const int icl::math::LLM::TRAIN_OUTPUTS = 4 [static]

training flag for updating output weights

const int icl::math::LLM::TRAIN_SIGMAS = 2 [static]

training flag for updating input sigmas


The documentation for this class was generated from the following file:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines