contextual_robustness

Submodules

contextual_robustness.base module

class contextual_robustness.base.Techniques(value)[source]

Bases: enum.Enum

Verification techniques enum

FORMAL = 'formal'
TEST = 'test'

contextual_robustness.datasets module

contextual_robustness.datasets.load_cifar() → Tuple[numpy.array, numpy.array, numpy.array, numpy.array, pandas.core.frame.DataFrame][source]

Loads the CIFAR dataset

Returns

(X_train, Y_train, X_test, Y_test, labels)

Return type

typing.Tuple[np.array, np.array, np.array, np.array, pd.DataFrame]

contextual_robustness.datasets.load_gtsrb() → Tuple[numpy.array, numpy.array, numpy.array, numpy.array, pandas.core.frame.DataFrame][source]

Loads a subset of classes from the GTSRB dataset (classes = 1,2,3,4,5,7,8)

Returns

(X_train, Y_train, X_test, Y_test, labels)

Return type

tuple[np.array, np.array, np.array, np.array, pd.DataFrame]

contextual_robustness.formal module

class contextual_robustness.formal.ContextualRobustnessFormal(model_path: str = '', model_name: str = '', model_args: dict = {}, transform_fn: callable = <function ContextualRobustnessFormal.<lambda>>, transform_args: dict = {}, transform_name: str = '', X: numpy.array = array([], dtype=float64), Y: numpy.array = array([], dtype=float64), sample_indexes: list = [], eps_lower: float = 0.0, eps_upper: float = 1.0, eps_interval: float = 0.002, marabou_options: dict = {'verbosity': 0}, verbosity: int = 0)[source]

Bases: contextual_robustness.base._BaseContextualRobustness

Class for ContextualRobustness ‘Formal Verification’ analysis

Parameters

  • model_path (str, optional) – Path to saved tensorflow model. Defaults to ‘’.

  • model_name (str, optional) – Name of model. Defaults to ‘’.

  • model_args (dict, optional) – Args passed to Marabou to load network. Defaults to dict().

  • X (np.array, optional) – The images. Defaults to np.array([]).

  • Y (np.array, optional) – Labels for images (onehot encoded). Defaults to np.array([]).

  • sample_indexes (list[int], optional) – List of indexes to test from X. Defaults to [].

  • transform_fn (callable, optional) – The image transform function (required args: x, epsilon). Defaults to lambda x:x.

  • transform_args (dict, optional) – Additional arguments passed to transform_fn. Defaults to dict().

  • transform_name (str, optional) – Name of transform. Defaults to ‘’.

  • eps_lower (float, optional) – Min possible epsilon. Defaults to 0.0.

  • eps_upper (float, optional) – Max possible epsilon. Defaults to 1.0.

  • eps_interval (float, optional) – Step size between possible epsilons. Defaults to 0.002.

  • marabou_options (dict, optional) – Additional MarabouOptions. Defaults to dict(verbosity=0).

  • verbosity (int, optional) – Amount of logging (0-4). Defaults to 0.

Returns

the ContextualRobustnessFormal object

Return type

ContextualRobustness

property accuracy

Gets the accuracy (optionally for a single class using “get_accuracy”)

Parameters

class_index (int, optional) – Index of class to get accuracy for. Defaults to None.

Returns

Accuracy of model on the samples (for a particular class if class_index is supplied)

Return type

float

analyze(epsilons_outpath: str = './epsilons.csv', counterexamples_outpath: str = './counterexamples.p') → ContextualRobustness

Run analysis on the model & transform; Generates results csv and counterexamples pickle.

Parameters

  • epsilons_outpath (str, optional) – Path to csv file containing results. Defaults to ‘./epsilons.csv’.

  • counterexamples_outpath (str, optional) – Path to pickle containing counterexamples. Defaults to ‘./counterexamples.p’.

Returns

the object (self)

Return type

ContextualRobustness

property classes

classes property

Returns

list of integers representing classes in dataset

Return type

list

property counterexamples

counterexamples property

Returns

np.array]: counterexamples for each image (e.g. {‘image1’: np.array([…]), …})

Return type

dict[str

property dataset

dataset property

Returns

tuple containing X and Y

Return type

tuple[np.array, np.array]

get_accuracy(class_index: Optional[int] = None) → float

Gets the accuracy (optionally for a single class using “get_accuracy”)

Parameters

class_index (int, optional) – Index of class to get accuracy for. Defaults to None.

Returns

Accuracy of model on the samples (for a particular class if class_index is supplied)

Return type

float

get_counterexample(x_index: int) → numpy.array

Gets counterexample for an image by index

Parameters

x_index ([int]) – index of image in X

Returns

the counterexample (or ‘None’ if does not exist)

Return type

np.array

get_num_samples(class_index: Optional[int] = None) → int

Gets the number of samples under analysis (optionally for a single class using “get_num_samples”).

Parameters

class_index (int, optional) – Index of class to get samples for. Defaults to None.

Returns

Number of samples (for a particular class if class_index is supplied)

Return type

int

get_results(class_index: Optional[int] = None, sort_by: list = []) → pandas.core.frame.DataFrame

Gets result data (optionally for a single class, and optionally sorted by column using “get_results”)

Parameters

  • class_index (int, optional) – Index of class to get results for. Defaults to None.

  • sort_by (list[str], optional) – Sorts results by one or more columns. Defaults to [].

Returns

The results (for a single class if class_index is supplied, and sorted by sort_by)

Return type

pd.DataFrame

property image_shape

image_shape property

Returns

shape of images in X

Return type

tuple[int, int, int]

property image_size

image_size property

Returns

size of images (width, height)

Return type

tuple[int, int]

load_results(epsilons_path: str = '', counterexamples_path: str = '') → ContextualRobustness

Load previously saved results

Parameters

  • epsilons_path (str, optional) – Path to results csv file. Defaults to ‘’.

  • counterexamples_path (str, optional) – Path to counterexamples pickle. Defaults to ‘’.

Returns

the object (self)

Return type

ContextualRobustness

property model_name

model_name property

Returns

name of model

Return type

str

property n_pixels

n_pixels property

Returns

number of pixels in each image.

Return type

int

property num_samples

Gets the number of samples under analysis (optionally for a single class using “get_num_samples”).

Parameters

class_index (int, optional) – Index of class to get samples for. Defaults to None.

Returns

Number of samples (for a particular class if class_index is supplied)

Return type

int

property results

Gets result data (optionally for a single class, and optionally sorted by column using “get_results”)

Parameters

  • class_index (int, optional) – Index of class to get results for. Defaults to None.

  • sort_by (list[str], optional) – Sorts results by one or more columns. Defaults to [].

Returns

The results (for a single class if class_index is supplied, and sorted by sort_by)

Return type

pd.DataFrame

save_counterexample(x_index: int, counterexample: numpy.array)

Saves the counterexample for an image

Parameters

  • x_index ([int]) – index of image

  • counterexample ([np.array]) – the counterexample to save

property technique

technique property

Returns

verification technique (Techniques.FORMAL)

Return type

Techniques

property transform_name

transform_name property

Returns

name of transform

Return type

str

contextual_robustness.reporting module

class contextual_robustness.reporting.ContextualRobustnessReporting[source]

Bases: object

Class containing ContextualRobustness reporting functions

static generate_accuracy_report_plot(cr_objects: Sequence[ContextualRobustness], outfile: str = './accuracy-report.png', linestyles: tuple = (), axis_fontsize: int = 12, legend_fontsize: int = 14, fontfamily: str = 'serif', fontweight: str = 'ultralight', legend_loc: str = 'best', usetex: bool = True)[source]

Plots epsilon/accuracy for a given transform on multiple models and saves as png

Parameters

  • cr_objects (typing.Sequence[ContextualRobustness]) – list of ContextualRobustness objects

  • outfile (str, optional) – Output file path. Defaults to ‘./accuracy-report.png’.

  • linestyles (tuple, optional) – [description]. Defaults to ().

  • axis_fontsize (int, optional) – Fontsize for axis text. Defaults to 12.

  • legend_fontsize (int, optional) – Fontsize for axis text. Defaults to 14.

  • fontfamily (str, optional) – Fontfamily for text. Defaults to ‘serif’.

  • fontweight (str, optional) – Fontweight for text. Defaults to ‘ultralight’.

  • legend_loc (str, optional) – Location of legend. Defaults to ‘best’.

  • usetex (bool, optional) – Use latex for text. Defaults to True.

static generate_class_accuracy_plot(cr: ContextualRobustness, outfile: str = './class-accuracy.png', axis_fontsize: int = 12, legend_fontsize: int = 14, fontfamily: str = 'serif', fontweight: str = 'ultralight', legend_loc: str = 'best', usetex: bool = True)[source]

Plots accuracy of each class at various epsilons for a model/transform and saves as png

Parameters

  • cr (ContextualRobustness) – ContextualRobustness object

  • outfile (str, optional) – Output file path. Defaults to ‘./counterexamples.png’.

  • axis_fontsize (int, optional) – Fontsize for axis text. Defaults to 12.

  • legend_fontsize (int, optional) – Fontsize for axis text. Defaults to 14.

  • fontfamily (str, optional) – Fontfamily for text. Defaults to ‘serif’.

  • fontweight (str, optional) – Fontweight for text. Defaults to ‘ultralight’.

  • legend_loc (str, optional) – Location of legend. Defaults to ‘best’.

  • usetex (bool, optional) – Use latex for text. Defaults to True.

static generate_counterexamples_plot(cr: ContextualRobustness, outfile: str = './counterexamples.png', dpi: int = 144, show_labels: bool = True, label_fontsize: int = 12, fontfamily: str = 'serif', fontweight: str = 'ultralight', usetex: bool = True)[source]

Plots counterexamples for a model/transform and saves as png

Parameters

  • cr (ContextualRobustness) – ContextualRobustness object

  • outfile (str, optional) – Output file path. Defaults to ‘./counterexamples.png’.

  • dpi (int, optional) – Dots per inch. Defaults to 144.

  • show_labels (bool, optional) – Show the class labels? Defaults to True.

  • label_fontsize (int, optional) – Fontsize of class labels. Defaults to 12.

  • fontfamily (str, optional) – Fontfamily for text. Defaults to ‘serif’.

  • fontweight (str, optional) – Fontfamily for text. Defaults to ‘ultralight’.

  • usetex (bool, optional) – Use latex for text. Defaults to True.

static generate_epsilons_plot(cr: ContextualRobustness, outfile: str = 'epsilons.png', xlabel: str = '', ylabel: str = 'epsilon', axis_fontsize: int = 24, fontfamily: str = 'serif', fontweight: str = 'ultralight', usetex: bool = True)[source]

Plots epsilons by class for a model/transform and saves as png

Parameters

  • cr (ContextualRobustness) – ContextualRobustness object

  • outfile (str, optional) – Output file path. Defaults to ‘epsilons.png’.

  • xlabel (str, optional) – x axis label. Defaults to ‘’.

  • ylabel (str, optional) – y axis label. Defaults to ‘epsilon’.

  • axis_fontsize (int, optional) – Fontsize for axis text. Defaults to 24.

  • fontfamily (str, optional) – Fontfamily for text. Defaults to ‘serif’.

  • fontweight (str, optional) – Fontfamily for text. Defaults to ‘ultralight’.

  • usetex (bool, optional) – Use latex for text. Defaults to True.

contextual_robustness.test module

class contextual_robustness.test.ContextualRobustnessTest(model_path: str = '', model_name: str = '', X: numpy.array = array([], dtype=float64), Y: numpy.array = array([], dtype=float64), sample_indexes: list = [], transform_fn: callable = <function ContextualRobustnessTest.<lambda>>, transform_args: dict = {}, transform_name: str = '', eps_lower: float = 0.0, eps_upper: float = 1.0, eps_interval: float = 0.002, verbosity: int = 0)[source]

Bases: contextual_robustness.base._BaseContextualRobustness

Class for ContextualRobustness ‘Test Based’ analysis

Parameters

  • model_path (str, optional) – Path to saved tensorflow model. Defaults to ‘’.

  • model_name (str, optional) – Name of model. Defaults to ‘’.

  • X (np.array, optional) – The images. Defaults to np.array([]).

  • Y (np.array, optional) – Labels for images (onehot encoded). Defaults to np.array([]).

  • sample_indexes (list[int], optional) – List of indexes to test from X. Defaults to [].

  • transform_fn (callable, optional) – The image transform function (required args: x, epsilon). Defaults to lambda x:x.

  • transform_args (dict, optional) – Additional arguments passed to transform_fn. Defaults to dict().

  • transform_name (str, optional) – Name of transform. Defaults to ‘’.

  • eps_lower (float, optional) – Min possible epsilon. Defaults to 0.0.

  • eps_upper (float, optional) – Max possible epsilon. Defaults to 1.0.

  • eps_interval (float, optional) – Step size between possible epsilons. Defaults to 0.002.

  • verbosity (int, optional) – Amount of logging (0-4). Defaults to 0.

Returns

the ContextualRobustnessTest object

Return type

ContextualRobustness

property accuracy

Gets the accuracy (optionally for a single class using “get_accuracy”)

Parameters

class_index (int, optional) – Index of class to get accuracy for. Defaults to None.

Returns

Accuracy of model on the samples (for a particular class if class_index is supplied)

Return type

float

analyze(epsilons_outpath: str = './epsilons.csv', counterexamples_outpath: str = './counterexamples.p') → ContextualRobustness

Run analysis on the model & transform; Generates results csv and counterexamples pickle.

Parameters

  • epsilons_outpath (str, optional) – Path to csv file containing results. Defaults to ‘./epsilons.csv’.

  • counterexamples_outpath (str, optional) – Path to pickle containing counterexamples. Defaults to ‘./counterexamples.p’.

Returns

the object (self)

Return type

ContextualRobustness

property classes

classes property

Returns

list of integers representing classes in dataset

Return type

list

property counterexamples

counterexamples property

Returns

np.array]: counterexamples for each image (e.g. {‘image1’: np.array([…]), …})

Return type

dict[str

property dataset

dataset property

Returns

tuple containing X and Y

Return type

tuple[np.array, np.array]

get_accuracy(class_index: Optional[int] = None) → float

Gets the accuracy (optionally for a single class using “get_accuracy”)

Parameters

class_index (int, optional) – Index of class to get accuracy for. Defaults to None.

Returns

Accuracy of model on the samples (for a particular class if class_index is supplied)

Return type

float

get_counterexample(x_index: int) → numpy.array

Gets counterexample for an image by index

Parameters

x_index ([int]) – index of image in X

Returns

the counterexample (or ‘None’ if does not exist)

Return type

np.array

get_num_samples(class_index: Optional[int] = None) → int

Gets the number of samples under analysis (optionally for a single class using “get_num_samples”).

Parameters

class_index (int, optional) – Index of class to get samples for. Defaults to None.

Returns

Number of samples (for a particular class if class_index is supplied)

Return type

int

get_results(class_index: Optional[int] = None, sort_by: list = []) → pandas.core.frame.DataFrame

Gets result data (optionally for a single class, and optionally sorted by column using “get_results”)

Parameters

  • class_index (int, optional) – Index of class to get results for. Defaults to None.

  • sort_by (list[str], optional) – Sorts results by one or more columns. Defaults to [].

Returns

The results (for a single class if class_index is supplied, and sorted by sort_by)

Return type

pd.DataFrame

property image_shape

image_shape property

Returns

shape of images in X

Return type

tuple[int, int, int]

property image_size

image_size property

Returns

size of images (width, height)

Return type

tuple[int, int]

load_results(epsilons_path: str = '', counterexamples_path: str = '') → ContextualRobustness

Load previously saved results

Parameters

  • epsilons_path (str, optional) – Path to results csv file. Defaults to ‘’.

  • counterexamples_path (str, optional) – Path to counterexamples pickle. Defaults to ‘’.

Returns

the object (self)

Return type

ContextualRobustness

property model_name

model_name property

Returns

name of model

Return type

str

property n_pixels

n_pixels property

Returns

number of pixels in each image.

Return type

int

property num_samples

Gets the number of samples under analysis (optionally for a single class using “get_num_samples”).

Parameters

class_index (int, optional) – Index of class to get samples for. Defaults to None.

Returns

Number of samples (for a particular class if class_index is supplied)

Return type

int

property results

Gets result data (optionally for a single class, and optionally sorted by column using “get_results”)

Parameters

  • class_index (int, optional) – Index of class to get results for. Defaults to None.

  • sort_by (list[str], optional) – Sorts results by one or more columns. Defaults to [].

Returns

The results (for a single class if class_index is supplied, and sorted by sort_by)

Return type

pd.DataFrame

save_counterexample(x_index: int, counterexample: numpy.array)

Saves the counterexample for an image

Parameters

  • x_index ([int]) – index of image

  • counterexample ([np.array]) – the counterexample to save

property technique

technique property

Returns

verification technique (Techniques.TEST)

Return type

Techniques

transform_image(x: numpy.array, epsilon: float) → numpy.array[source]

Transforms an image using transform_fn

Parameters

  • x (np.array) – The image

  • epsilon (float) – amount of transform

Returns

The transformed image

Return type

np.array

property transform_name

transform_name property

Returns

name of transform

Return type

str

contextual_robustness.transforms module

contextual_robustness.transforms.encode_haze(network: <Mock name='mock.MarabouNetwork' id='140238308817232'>, image: numpy.array, epsilon: float, output_index: int) → <Mock name=’mock.MarabouNetwork’ id=’140238308817232’>[source]

Encodes a haze transformation as a Marabou input query

Parameters

  • network (MarabouNetwork) – the MarabouNetwork object

  • image (np.array) – The input image

  • epsilon (float) – Amount of transform

  • output_index (int) – Target output node (for the expected class)

Returns

The MarabouNetwork object with the encoded input query

Return type

MarabouNetwork

contextual_robustness.transforms.encode_linf(network: <Mock name='mock.MarabouNetwork' id='140238308817232'>, image: numpy.array, epsilon: float, output_index: int) → <Mock name=’mock.MarabouNetwork’ id=’140238308817232’>[source]

Encodes a linear perturbation as a Marabou input query

Parameters

  • network (MarabouNetwork) – the MarabouNetwork object

  • image (np.array) – The input image

  • epsilon (float) – Amount of transform

  • output_index (int) – Target output node (for the expected class)

Returns

The MarabouNetwork object with the encoded input query

Return type

MarabouNetwork

contextual_robustness.transforms.gaussianblureps(image: numpy.array, epsilon: float, kernelSize: int = 17, scaling: int = 20) → numpy.array[source]

Applies gaussian blur transform to input image

Parameters

  • image (np.array) – The input image

  • epsilon (float) – Amount of transform

  • kernelSize (int, optional) – The kernel size. Defaults to 17.

  • scaling (int, optional) – Scaling of the blur transform. Defaults to 20.

Returns

The transformed image

Return type

np.array

contextual_robustness.transforms.haze(image: numpy.array, epsilon: float) → numpy.array[source]

Applies haze transform to an image

Parameters

  • image (np.array) – The input image

  • epsilon (float) – amount of transform

Returns

image with haze

Return type

[np.array]

contextual_robustness.transforms.increase_contrast(image: numpy.array, epsilon: float, tg_min: float = 0.0, tg_max: float = 1.0) → numpy.array[source]

Increases the contrast of the input image

Parameters

  • image (np.array) – The input image

  • epsilon (float) – Amount of transform

  • tg_min (float, optional) – Min value of image scaling. Defaults to 0.0.

  • tg_max (float, optional) – Max value of image scaling. Defaults to 1.0.

Returns

The transformed image

Return type

np.array

contextual_robustness.utils module

class contextual_robustness.utils.Timer(autostart: bool = False)[source]

Bases: object

A simple timer class

property elapsed

elapsed time property (as string using get_elapsed(as_string=True))

Parameters

as_string (bool, optional) – returns as string with unit (ms). Defaults to False.

Returns

milliseconds between start_time and end_time

Return type

int

end() → int[source]

stops the timer

Returns

elapsed time (ms)

Return type

int

property end_time

end_time property

Returns

the end_time (ms since 1970)

Return type

int

get_elapsed(as_string: bool = False) → int[source]

elapsed time property (as string using get_elapsed(as_string=True))

Parameters

as_string (bool, optional) – returns as string with unit (ms). Defaults to False.

Returns

milliseconds between start_time and end_time

Return type

int

start() → object[source]

start the timer

Returns

the Timer object

Return type

object

property start_time

start_time property

Returns

the start_time (ms since 1970)

Return type

int

contextual_robustness.utils.normalize(X: numpy.array) → numpy.array[source]

normalizes image values between 0.0 and 1.0

Parameters

X (np.array) – array of images

Returns

normalized images

Return type

np.array

contextual_robustness.utils.parse_indexes(indexes_list: List[str], sort=True) → List[int][source]

Parses mixed list of integers and ranges from CLI into a discret list of integers.

Parameters

indexes_list (list, optional) – List of strings of mixed ints and/or ranges (e.g. [‘1’, ‘2’, ‘5-7’, ‘10-15’]). Defaults to [].

Returns

Discret list of unique integers

Return type

list[int]

contextual_robustness.utils.remove_softmax_activation(model_path: str, save_path: str = '') → tensorflow.python.keras.engine.training.Model[source]

Prepares a classifier with softmax activation for verification by removing the softmax activation function from the output layer.

Parameters

  • model_path (str) – Path to model

  • save_path (str, optional) – Path where new model is saved. Defaults to ‘’.

Returns

The modified tensorflow Model object

Return type

tf.keras.Model

contextual_robustness.utils.resize_image(image: numpy.array, size: Tuple[int, int]) → numpy.array[source]

resizes an image

Parameters

  • image (np.array) – the original image

  • size (typing.Tuple[int, int]) – size of resized image (width, height)

Returns

resized image

Return type

np.array

contextual_robustness.utils.set_df_dtypes(df: pandas.core.frame.DataFrame, dtypes: dict) → pandas.core.frame.DataFrame[source]

Sets datatypes for specified columns of DataFrame

Parameters

  • df (pd.DataFrame) – The DataFrame

  • dtypes (dict) – Dictionary of datatypes (e.g. {‘col’:type, …})

Returns

The updated DataFrame

Return type

pd.DataFrame

contextual_robustness.utils.softargmax(y: numpy.array) → numpy.array[source]

Applies softmax & argmax to emulate the softmax output layer of a tensorflow model

Parameters

y (np.array) – Logits layer output

Returns

onehot encoded prediction (e.g. [0,0,1,0])

Return type

np.array