DAGs Encoding#

Main Objects#

DRAGON search space is base on Directed Acyclic Graphs (DAGs) implemented using an adjacency matrix combined with a set of node. The nodes represent the network layers and the edges the connexion between them.

class Brick(nn.Module)[source]

Bases: Module

The Meta class Brick serves as a basis to incorporate the nn.Module layers from PyTorch into DRAGON. In addition to the __init__ and forward functions, they should have a method to modify the layer given an input shape. The **args correspond to the layer hyperparameters.

Parameters:: input_shape (tuple) – Shape of the input tensor.

foward(X)[source]

forward(X)

Forward pass.

Parameters:: X (torch.Tensor) – Input tensor.

modify_operation(input_shape)[source]

Modify the operation so it can take a tensor of shape input_shape as input.

Parameters:: input_shape (tuple) – Shape of the input tensor.

class Node(nn.Module)[source]

Bases: Module

The class Node is the implementation of a DAG node. Each node is made of a combiner, an operation and an activation function. The operation is parametrized by a set of hyperparameters.

Parameters:

combiner (str) – Name of the combiner. The only combiner implemented for now within DRAGON are: ‘add’, ‘concat’ and ‘mul’
operation (Brick) – Operation that will be performed within the node.
hp (dict) – Dictionary containing the hyperparameters. The keys name should match the arguments of the operation class __init__ function.
activation (nn.Module, default=nn.Identity()) – Activation function.
input_comp (['Pad', 'Crop'], default='Pad') – Defines how the combiner will compute the input shape. When set to ‘Pad’, the maximum input shape from all the incoming tensors will be taken. When set to ‘Crop’ the mean input shape will be taken.

Examples

>>> import torch.nn as nn
>>> from dragon.search_space.bricks import MLP
>>> from dragon.search_space.dragon_variables import Node
>>> print(Node(combiner="add", operation=MLP, hp={"out_channels": 10}, activation=nn.ReLU()))
(combiner) add -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 10} -- (activation) ReLU() --

copy()[source]

Creates an new Node variable which is a copy of this one.

Returns:: new_node – Copy of the actual Node.
Return type:: Node

set_operation(input_shapes, device=None)[source]

Initialize the operation using the new input shapes. First, the global input shape of the operation is computed, using the combiner type and the input_comp attribute Then, the operation is initialized with the global input shape and the hyperparameters. The operation parameters are modified with the xavier_uniform initialization. Finally, the node ouput shape is computed.

Parameters:

input_shapes (list, tuple or int) – Input shapes of the multiple (or single) input vectors of the node.
device (str, default=None) – Device on which the node operation should be computed.

compute_input_shape(X, h=None)[source]

Compute the global input shape for the operation, given the (possibly) multiple input shapes. The global shape depends on the combiner type and the value of self.input_comb.

Parameters:: input_shapes (list) – List containing the input shapes of the different input tensors.
Return type:: tuple

combine(X)[source]

Use the combiner to combine the input vectors. First the vectors are modified to have the global input shape using the self.padding function. Then they are combined by addition, multiplication or concatenation.

Parameters:: X (list) – List containing the input tensors.
Return type:: torch.Tensor

padding(X, start=-1, pad_start=())[source]

Modify the input tensors gathered in X so they all have the global input shape. The padding is performed over all dimensions for the ‘add’ and ‘mul’ combiners, but not on the last one for the ‘concat’ combiner.

Parameters:

X (list or torch.Tensor) – List containing the input tensors.
start (int, default=-1) – Dimension where to start the padding. It depends on the combiner.
pad_start (tuple, default=()) – Default padding over the last dimension. It depends on the combiner.

Returns:

pad_X – List containing the tensors with the right shape.

Return type:

list

compute_output_shape()[source]

Compute the output shape of the node. A fake vector is created with a shape equals to the global input shape. This fake vector is processed by the operation. The output vector shape will be the node output shape.

Returns:: shape – The node output shape.
Return type:: tuple

modification(combiner=None, name=None, hp=None, input_shapes=None, device=None)[source]

Modify the node. The modifications can be applied to the combiner, the operation, the operation’s hyperparameters or the input shapes. The values set to None will stay unchanged. If the operation and the hyperparameters do not change, the operation is just modified. Otherwise a new one will be created.

Parameters:

combiner (str, default=None) – Name of the new combiner.
operation (Brick, default=None) – New operation that will be performed within the node.
hp (dict, default=None) – Dictionary containing the new hyperparameters. The keys name should match the arguments of the operation class __init__ function.
input_shape (list, tuple or int, default=None) – List of the new input shapes.
device (str, default=None) – Name of the device where the node is computed.

modify_operation(input_shape)[source]

Modify the operation so it can take as input a tensor of shape input_shape.

Parameters:: input_shape (tuple) – New input shape.

set(input_shapes)[source]

Initialize or modify the node with the incoming shapes `input_shape`s .

Parameters:: input_shapes (list, tuple or int) – Input shapes of the multiple (or single) input vectors of the node.

forward(X, h=None)[source]

Forward pass of the layer. The inputs are first combined by the combiner, then processed by the operation and the activation function.

Parameters:

X (torch.Tensor or list) – Input tensor or list of input tensors.
h (torch.Tensor, default=None) – Hidden state, used in the case of recurrent layer.

Returns:

X or (X,h) – Processed tensor(s).

Return type:

torch.Tensor

load_state_dict(state_dict, **kwargs)[source]

Copy parameters and buffers from state_dict into this module and its descendants.

If strict is True, then the keys of state_dict must exactly match the keys returned by this module’s state_dict() function.

Warning

If assign is True the optimizer must be created after the call to load_state_dict unless get_swap_module_params_on_conversion() is True.

Parameters:

state_dict (dict) – a dict containing parameters and persistent buffers.
strict (bool, optional) – whether to strictly enforce that the keys in state_dict match the keys returned by this module’s state_dict() function. Default: True
assign (bool, optional) – When False, the properties of the tensors in the current module are preserved while when True, the properties of the Tensors in the state dict are preserved. The only exception is the requires_grad field of Default: ``False`

Returns:

missing_keys is a list of str containing any keys that are expected
by this module but missing from the provided state_dict.
unexpected_keys is a list of str containing the keys that are not
expected by this module but present in the provided state_dict.

Return type:

NamedTuple with missing_keys and unexpected_keys fields

Note

If a parameter or buffer is registered as None and its corresponding key exists in state_dict, load_state_dict() will raise a RuntimeError.

fill_adj_matrix(matrix)[source]

Add random edges into an adjacency matrix in case it contains orphan nodes (no incoming connection) or nodes having no outgoing connection. Except from the first node, all nodes should have at least one incoming connection, meaning the corresponding column should not sum to zero. Except from the last node, all nodes should have at least one outgoing connection, meaning the corresponding row should not sum to zero.

Parameters:: matrix (np.array) – Adjacency matrix from a DAG that may contain orphan nodes.
Returns:: matrix – Adjacency matrix from a DAG that does not contain orphan nodes.
Return type:: np.array

DRAGON variables#

Like the int, float, categorial and array based elements are embedded using Variable based classes, variables have also been implemented for the new objects: AdjMatrix, Node and Brick.

class HpVar(Variable)[source]

Bases: Variable

The class HpVar defines Variable which represent a node operation. The operation can be a Constant or a CatVar, where the values inherit from the Brick class. If the operation is represented by a Constant, the multiple operations should share the same hyperparameters.

Parameters:

label (str) – Name of the variable.
operation (Constant or CatVar) – One or several candidate operations encoded as Brick variable. If operation is a CatVar, the multiple operations should share the same hyperparameters.
hyperparameters (dict) – Dictionary of hyperparameters which inherit from Variables (for example IntVar for a number of channels or FloatVar for a dropout rate).

Examples

>>> from dragon.search_space.bricks import MLP
>>> from dragon.search_space.zellij_variables import Constant, IntVar
>>> from dragon.search_space.dragon_variables import HpVar
>>> mlp = Constant("Mlp operation", MLP)
>>> hp = {"out_channels": IntVar("out_channels", 1, 10)}
>>> mlp_var = HpVar("MLP var", mlp, hyperparameters=hp)
>>> mlp_var.random()
[<class 'dragon.search_space.bricks.basics.MLP'>, {'out_channels': 9}]

>>> from dragon.search_space.bricks import LayerNorm1d, BatchNorm1d
>>> from dragon.search_space.zellij_variables import CatVar
>>> norm = CatVar("1d norm layers", features=[LayerNorm1d, BatchNorm1d])
>>> norm_var = HpVar("Norm var", norm, hyperparameters={})
>>> norm_var.random()
[<class 'dragon.search_space.bricks.normalization.BatchNorm1d'>, {}]

random(size=1)[source]

Create random operation. First, if the operation is a CatVar, an operation is randomly selected among the different possibilities. Then, one random value per hyperparameter is drawn.

Parameters:: size (int, default=1) – Number of draws.
Returns:: matrices – List containing the randomly created operations, or a single operation if size=1.
Return type:: list or AdjMatrix

isconstant()[source]

Returns:: out
Return type:: False

class NodeVariable(Variable)[source]

Bases: Variable

The class NodeVariable defines Variable which represent DAGs nodes by creating objects from the Node class.

Parameters:

label (str) – Name of the variable.
operations (DynamicBlock) – DynamicBlock containing Variable corresponding to the candidate operations.
init_complexity (int) – Maximum number of nodes that the randomly created DAGs should have.

Examples

>>> from dragon.search_space.dragon_variables import NodeVariable, HpVar
>>> from dragon.search_space.bricks import MLP
>>> from dragon.search_space.zellij_variables import Constant, IntVar, CatVar
>>> from dragon.search_space.bricks_variables import activation_var
>>> combiner = CatVar("Combiner", features = ['add', 'mul'])
>>> operation = HpVar("Operation", Constant("Mlp operation", MLP), hyperparameters={"out_channels": IntVar("out_channels", 1, 10)})
>>> node = NodeVariable(label="Node variable",
...                 combiner=combiner,
...                 operation=operation,
...                 activation_function=activation_var("Activation"))
>>> node.random()
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 2} -- (activation) SiLU() --

random(size=1)[source]

Create random nodes. The combiner, the operation and the activation function are sequentally randomly selected.

Parameters:: size (int, default=1) – Number of draws.
Returns:: matrices – List containing the randomly created nodes, or a single node if size=1.
Return type:: list or Node

isconstant()[source]

Returns:: out – Return False, a dynamic block cannot be constant. (It is a binary)
Return type:: False

class EvoDagVariable(Variable)[source]

Bases: Variable

The class EvoDagVariable defines Variable which represent Directed Acyclic Graph by creating objects from the AdjMatrix class. The candidate operations should be gathered within a DynamicBlock. The maximum size of this DynamicBlock will set the graph maximum number of nodes.

Parameters:

label (str) – Name of the variable.
operations (DynamicBlock) – DynamicBlock containing Variable corresponding to the candidate operations.
init_complexity (int) – Maximum number of nodes that the randomly created DAGs should have.

Examples

>>> from dragon.search_space.dragon_variables import HpVar, NodeVariable, EvoDagVariable
>>> from dragon.search_space.bricks import MLP, MaxPooling1D, AVGPooling1D
>>> from dragon.search_space.zellij_variables import Constant, IntVar, CatVar, DynamicBlock
>>> from dragon.search_space.bricks_variables import activation_var
>>> mlp = HpVar("Operation", Constant("MLP operation", MLP), hyperparameters={"out_channels": IntVar("out_channels", 1, 10)})
>>> pooling = HpVar("Operation", CatVar("Pooling operation", [MaxPooling1D, AVGPooling1D]), hyperparameters={"pool_size": IntVar("pool_size", 1, 5)})
>>> candidates = NodeVariable(label = "Candidates",
...                         combiner=CatVar("Combiner", features=['add', 'concat']),
...                         operation=CatVar("Candidates", [mlp, pooling]),
...                         activation_function=activation_var("Activation"))
>>> operations = DynamicBlock("Operations", candidates, repeat=5)
>>> dag = EvoDagVariable(label="DAG", operations=operations)
>>> dag.random()
NODES: [
(combiner) add -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) Identity() -- ,
(combiner) add -- (name) <class 'dragon.search_space.bricks.pooling.MaxPooling1D'> -- (hp) {'pool_size': 2} -- (activation) Sigmoid() -- ,
(combiner) concat -- (name) <class 'dragon.search_space.bricks.pooling.MaxPooling1D'> -- (hp) {'pool_size': 3} -- (activation) ELU(alpha=1.0) -- ,
(combiner) add -- (name) <class 'dragon.search_space.bricks.pooling.AVGPooling1D'> -- (hp) {'pool_size': 4} -- (activation) ReLU() -- ] | MATRIX:[[0, 1, 1, 1], [0, 0, 1, 1], [0, 0, 0, 1], [0, 0, 0, 0]]

random(size=1)[source]

Create random DAGs. First, a list of random nodes is creating, with a size lower than the :code: complexity attribute. The first element of this list will always be an Identity layer. Then, the adjacency matrix is created as an upper-triangle matrix, with the same size as the list. This adjacency matrix is corrected using the :code: fill_adj_matrix function to prevent nodes from having no incoming or outgoing connections.

Parameters:: size (int, default=1) – Number of draws.
Returns:: matrices – List containing the randomly created DAGs, or a single DAG if size=1.
Return type:: list or AdjMatrix

isconstant()[source]

Returns:: out – Return False, a dynamic block cannot be constant. (It is a binary)
Return type:: False

DAGs Encoding

Contents

DAGs Encoding#

Main Objects#

DRAGON variables#