grb.evaluator

grb.evaluator.evaluator

Evaluator Module for Unified Evaluation of Attacks vs. Defenses.

class grb.evaluator.evaluator.AttackEvaluator(dataset, build_model, device='cpu')[source]

Bases: object

Evaluator used to evaluate the attack performance on a dataset across different models.

Parameters

  • dataset (grb.dataset.Dataset or grb.dataset.CustomDataset) – GRB supported dataset.

  • build_model (func) – Function that builds a model with specific configuration.

  • device (str, optional) – Device used to host data. Default: cpu.

eval(model, adj, features, adj_norm_func=None)[source]

Evaluate attack results on a single model.

Parameters

  • model (torch.nn.module) – Model implemented based on torch.nn.module.

  • adj (scipy.sparse.csr.csr_matrix) – Adjacency matrix in form of N * N sparse matrix.

  • features (torch.FloatTensor) – Features in form of N * D torch float tensor.

  • adj_norm_func (func of utils.normalize, optional) – Function that normalizes adjacency matrix. Default: None.

Returns

test_score – The test score of the model on input adjacency matrix and features.

Return type

float

eval_attack(model_dict, adj_attack, features_attack, verbose=False)[source]

Evaluate attack results on single/multiple model(s).

Parameters

  • model_dict (dict) – Dictionary in form of {'model_name', 'model_path'}. model_name should be compatible with build_model func.

  • adj_attack (scipy.sparse.csr.csr_matrix) – Adversarial adjacency matrix in form of N * N sparse matrix.

  • features_attack (torch.FloatTensor) – Features of nodes after attacks in form of N * D torch float tensor.

  • verbose (bool, optional) – Whether to display logs. Default: False.

Returns

test_score_dict – Dictionary in form of {'model_name', 'evaluation score'}.

Return type

dict

static eval_metric(test_score_sorted, metric_type='polynomial', order='a')[source]
Parameters

  • test_score_sorted – Array of sorted test scores.

  • metric_type (str, optional) – Type of metric. Default: polynomial.

  • order (str, optional) – Ascending order a or descending order d. Default: a.

Returns

final_score – Final general score across methods.

Return type

float

class grb.evaluator.evaluator.DefenseEvaluator(dataset, build_model, device='cpu')[source]

Bases: object

grb.evaluator.metric

Evaluation metrics

grb.evaluator.metric.eval_acc(pred, labels, mask=None)[source]

Accuracy metric for node classification.

Parameters

  • pred (torch.Tensor) – Output logits of model in form of N * 1.

  • labels (torch.LongTensor) – Labels in form of N * 1.

  • mask (torch.Tensor, optional) – Mask of nodes to evaluate in form of N * 1 torch bool tensor. Default: None.

Returns

acc – Node classification accuracy.

Return type

float

grb.evaluator.metric.eval_f1multilabel(pred, labels, mask=None)[source]

F1 score for multi-label node classification.

Parameters

  • pred (torch.Tensor) – Output logits of model in form of N * 1.

  • labels (torch.LongTensor) – Labels in form of N * 1.

  • mask (torch.Tensor, optional) – Mask of nodes to evaluate in form of N * 1 torch bool tensor. Default: None.

Returns

f1 – Average F1 score across different labels.

Return type

float

grb.evaluator.metric.eval_rocauc(pred, labels, mask=None)[source]

ROC-AUC score for multi-label node classification.

Parameters

  • pred (torch.Tensor) – Output logits of model in form of N * 1.

  • labels (torch.LongTensor) – Labels in form of N * 1.

  • mask (torch.Tensor, optional) – Mask of nodes to evaluate in form of N * 1 torch bool tensor. Default: None.

Returns

rocauc – Average ROC-AUC score across different labels.

Return type

float

grb.evaluator.metric.get_weights_arithmetic(n, w_1, order='a')[source]

Arithmetic weights for calculating weighted robust score.

Parameters

  • n (int) – Number of scores.

  • w_1 (float) – Initial weight of the first term.

  • order (str, optional) – a for ascending order, d for descending order. Default: a.

Returns

weights – List of weights.

Return type

list

grb.evaluator.metric.get_weights_polynomial(n, p=2, order='a')[source]

Arithmetic weights for calculating weighted robust score.

Parameters

  • n (int) – Number of scores.

  • p (float) – Power of denominator.

  • order (str, optional) – a for ascending order, d for descending order. Default: a.

Returns

weights_norms – List of normalized polynomial weights.

Return type

list