How can I create a learnable parameter or weight vector whose values are either 1 or -1

Question

I need to build a neural network which is known as learnable parameter or a weight vector. In that way, only one vector will be generated and multiplied with the data. I have created that as following:

from tensorflow.keras.layers import Layer
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model

class LearnableMultiplier(Layer):
    def __init__(self, **kwargs):
        super(LearnableMultiplier, self).__init__(**kwargs)

    def build(self, input_shape):
        self.kernel = self.add_weight(name='kernel',
                                      shape=(input_shape[-1],),
                                      initializer='glorot_uniform',
                                      trainable=True)

        super(LearnableMultiplier, self).build(input_shape)

    def call(self, inputs):
        return inputs * self.kernel

inputs = Input(shape=(64,))
multiplier = LearnableMultiplier()(inputs)  
model = Model(inputs=inputs, outputs=multiplier )

I need the vector learnable parameter defined above to be selected from values of either 1 or -1. I mean each value of that vector which is multiplied with my data can only be 1 or -1. Is that feasible? How can I do it?

Asked By: Sajjad

||

Source

Answer 1

Yes, it is feasible to constrain the learnable weight vector to be either 1 or -1. One way to achieve this is by using the sign function to convert the weights to either 1 or -1. You can modify your call function to apply the sign function to the weights, like this:

import tensorflow as tf
from tensorflow.keras.layers import Layer
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model

class LearnableMultiplier(Layer):
    def __init__(self, **kwargs):
        super(LearnableMultiplier, self).__init__(**kwargs)

    def build(self, input_shape):
        self.kernel = self.add_weight(name='kernel',
                                      shape=(input_shape[-1],),
                                      initializer='glorot_uniform',
                                      trainable=True)

        super(LearnableMultiplier, self).build(input_shape)

    def call(self, inputs):
        return inputs * tf.math.sign(self.kernel)

The tf.math.sign function will convert each element of the weight vector to either 1 or -1, depending on whether it is positive or negative. This will ensure that only 1 or -1 are used as the values for the weight vector during the multiplication.

Update: How about if I want to extend it to make every value selected from a list of different values, for example [1, -1, 0.5, -0.5], is that still feasible to be done?

To do this, you can replace the tf.math.sign function in the call method with a custom logic that selects the appropriate value from the list based on the sign of the kernel weight. Here is an example implementation:

import tensorflow as tf
from tensorflow.keras.layers import Layer

class LearnableMultiplier(Layer):
    def __init__(self, values, **kwargs):
        super(LearnableMultiplier, self).__init__(**kwargs)
        self.values = values

    def build(self, input_shape):
        self.kernel = self.add_weight(name='kernel',
                                      shape=(input_shape[-1],),
                                      initializer='glorot_uniform',
                                      trainable=True)

        super(LearnableMultiplier, self).build(input_shape)

    def call(self, inputs):
        signs = tf.math.sign(self.kernel)
        values = tf.gather(self.values, tf.cast((signs + 1) / 2, tf.int32))
        return inputs * values

The modified LearnableMultiplier layer takes an additional argument values, which is a list of values from which to select the learnable scalar factor. In the call method, the tf.math.sign function is replaced with a call to tf.gather that selects the appropriate value from the values list based on the sign of the kernel weight. Specifically, we first map the sign values from the range [-1, 1] to the range [0, 1] by adding 1 and dividing by 2, and then use the resulting integer indices to select the corresponding values from the list.

To use the modified layer in a Keras model, you can create an instance of the LearnableMultiplier class and pass it the values argument. For example:

import tensorflow as tf
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model

values = [1, -1, 0.5, -0.5]
inputs = Input(shape=(10,))
x = LearnableMultiplier(values=values)(inputs)
outputs = Dense(1)(x)
model = Model(inputs=inputs, outputs=outputs)

Answered By: Ekaba Bisong

How can I create a learnable parameter or weight vector whose values are either 1 or -1

Question:

Answers: