Quickstart#

[1]:

import warnings
warnings.filterwarnings('ignore')
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import openml
from sklearn.model_selection import train_test_split
import numpy as np
import os

Loading the dataset#

This tutorial introduces you to a complete machine learning pipeline to run DRAGON for a simple classification task on the PenDigits dataset.

[2]:

dataset = openml.datasets.get_dataset(32)
data, _, numerical, names = dataset.get_data()
data.head()

[2]:

	input1	input2	input3	input4	input5	input6	input7	input8	input9	input10	input11	input12	input13	input14	input15	input16	class
0	47.0	100.0	27.0	81.0	57.0	37.0	26.0	0.0	0.0	23.0	56.0	53.0	100.0	90.0	40.0	98.0	8
1	0.0	89.0	27.0	100.0	42.0	75.0	29.0	45.0	15.0	15.0	37.0	0.0	69.0	2.0	100.0	6.0	2
2	0.0	57.0	31.0	68.0	72.0	90.0	100.0	100.0	76.0	75.0	50.0	51.0	28.0	25.0	16.0	0.0	1
3	0.0	100.0	7.0	92.0	5.0	68.0	19.0	45.0	86.0	34.0	100.0	45.0	74.0	23.0	67.0	0.0	4
4	0.0	67.0	49.0	83.0	100.0	100.0	81.0	80.0	60.0	60.0	40.0	40.0	33.0	20.0	47.0	0.0	1

[3]:

X = data.drop('class', axis=1)
y = data[["class"]].astype(int)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0)
print(f"X_train: {X_train.shape}, y_train: {y_train.shape}, X_val: {X_val.shape}, y_val: {y_val.shape}")
print(f"Number of features: {X.shape[1]}, number of classes: {y.shape[1]}")

X_train: (7694, 16), y_train: (7694, 1), X_val: (3298, 16), y_val: (3298, 1)
Number of features: 16, number of classes: 1

[4]:

class CustomDataset(Dataset):
    def __init__(self, X, y):
        super().__init__()
        self.X = torch.FloatTensor(X.values)
        self.y = torch.LongTensor(y.values)
    def __len__(self):
        return self.X.shape[0]
    def __getitem__(self, index):
        return self.X[index], self.y[index]
train_set = CustomDataset(X_train, y_train)
val_set = CustomDataset(X_val, y_val)
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
val_loader = DataLoader(val_set, batch_size=1, shuffle=False)

Search Space#

In DRAGON the neural networks are represented as Directed Acyclic Graphs (DAGs), where each node represents nn.Module layer, and the edges are the connection between them. Within the package, the DAGs are encoded using their adjaceny matrix. To know more about these objects, refer to the Search Space section.

Meta-Architecture#

The meta-architecture $f$ should encompasses all the considered neural networks from the search space. To define a meta-architecture usable with DRAGON, we create a class that inherits from nn.Module. The __init__ function should take as input arguments defining a certain set of architecture and hyperparameters $\{\alpha, \theta\}$ . Then, the __forward__ function should take an input a variable $X$ and returns the prediction $\hat{y} = f_{\alpha,\theta}(X)$ .

The main bricks to build neural networks within the DRAGON package are dragon.search_space.cells.AdjMatrix and dragon.search_space.cells.Node. The AdjMatrix creates a DAG and the Node a simple layer. Both bricks can be used to create models adapted to the task at hand. In this case we stay very simple with a single AdjMatrix embodying the neural network and a final Node to get the right output shape.

Before the first training of a given neural network, the layers should be initialized with the right input shape.

[5]:

from dragon.search_space.cells import AdjMatrix, Node

class MetaArchi(nn.Module):
    def __init__(self, args, input_shape):
        super().__init__()
        # Number of features, here equals to 16
        self.input_shape = input_shape

        # We create the DAG using the WeightsAdjCell module
        assert isinstance(args['Dag'], AdjMatrix), f"The 'Dag' argument should be an 'AdjMatrix'. Got {type(args['Dag'])} instead."
        self.dag = args['Dag']
        self.dag.set(input_shape)

        # We set the final layer
        assert isinstance(args['Out'], Node), f"The 'Out' argument should be a 'Node'. Got {type(args['Node'])} instead."
        self.output = args["Out"]
        self.output.set(self.dag.output_shape)

    def forward(self, X):
        out = self.dag(X)
        return self.output(out)

    def save(self, path):
        if not os.path.exists(path):
            os.makedirs(path)
        full_path = os.path.join(path, "best_model.pth")
        torch.save(self.state_dict(), full_path)

Arguments#

The MetaArchi class takes as input a dictionnary args containing one search space configuration ${\alpha, \theta}$ . We now need to define what values $\alpha$ and $\theta$ can take. To keep it simple, we will only use nn.Linear and nn.Identity layers as candidate operations. Several ready-to-use candidate operations are available within dragon.search_space.bricks_variables. They are then used to build Directed Acyclic Graphs.

[6]:

from dragon.search_space.bricks_variables import mlp_var, identity_var, operations_var, mlp_const_var, dag_var, node_var

candidate_operations = operations_var("Candidate operations", size=10, candidates=[mlp_var("MLP"), identity_var("Identity")])
dag = dag_var("Dag", candidate_operations)
print(f'An example of a generated DAG: {dag.random()}')

# For the last layer, we do not want the number of output channels to be optimized. It should be equal to the number of classes.
out = node_var("Out", operation=mlp_const_var('Operation', 10), activation_function=nn.Softmax())

An example of a generated DAG: NODES: [
(combiner) add -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) Identity() -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) SiLU() -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) Tanh() -- ,
(combiner) concat -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 36} -- (activation) LeakyReLU(negative_slope=0.01) -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 378} -- (activation) Sigmoid() -- ,
(combiner) concat -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 170} -- (activation) SiLU() -- ] | MATRIX:[[0, 1, 1, 1, 0, 0], [0, 0, 1, 1, 0, 0], [0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 1], [0, 0, 0, 0, 0, 0]]

With the elements dag and out we can generate composants to build a neural network.

[7]:

args = {"Dag": dag.random(), "Out": out.random()}
model = MetaArchi(args, input_shape=(16,))

The final search space is an array which contain all the structural elements of the architecture as well as eventually other elements for the training for example.

[8]:

from dragon.search_space.zellij_variables import ArrayVar
from dragon.search_algorithm.zellij_neighborhoods import ArrayInterval

search_space = ArrayVar(dag, out, label="Search Space", neighbor=ArrayInterval())

Loss function#

The search algorithm should minimize a certain loss function. Typically, it consists in training the model on a train set and then validate it on a validation set.

[9]:

def train_model(model, data_loader):
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
    model.train()
    for _ in range(10):
        for X,y in data_loader:
            optimizer.zero_grad()
            y = y.squeeze()
            pred = model(X)
            loss = loss_fn(pred,y)
            loss.backward()
            optimizer.step()
    return model

def test_model(model, data_loader):
    loss_fn = nn.CrossEntropyLoss()
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
      for X, y in data_loader:
          y = y.squeeze(1)
          pred = model(X)
          loss = loss_fn(pred, y).item()
          test_loss += loss
          prediction = pred.argmax(axis=1)
          correct += (prediction == y).sum().item()
    accuracy = correct/ len(data_loader.dataset)
    return accuracy

def loss_function(args, *kwargs):
    labels = [e.label for e in search_space]
    args = dict(zip(labels, args))
    model = MetaArchi(args, input_shape=(16,))
    model = train_model(model, train_loader)
    accuracy = test_model(model, val_loader)
    return 1 - accuracy, model

loss, model = loss_function(search_space.random())
print(f'For a given argument, accuracy = ', np.round((1-loss)*100,2), '%')

For a given argument, accuracy =  10.28 %

Search Algorithm#

We can use several search algorithms to optimize the structures from the search space. Here, we take the example of the evolutionary algorithm.

[11]:

from dragon.search_algorithm.ssea import SteadyStateEA

search_algorithm = SteadyStateEA(search_space, n_iterations=50, population_size=10, selection_size=3, evaluation = loss_function, save_dir="save/test")
min_loss = search_algorithm.run()

2024-09-27 08:42:02,856 | WARNING | Install mpi4py if you want to use the distributed version.
2024-09-27 08:42:02,861 | INFO | The whole population has been created (size = 10), 10 have been randomy initialized.
2024-09-27 08:42:02,862 | INFO | We start by evaluating the whole population (size=10)
2024-09-27 08:42:05,709 | INFO | Best found ! 0.9120679199514857 < inf
2024-09-27 08:42:24,220 | INFO | Best found ! 0.04790782292298368 < 0.9120679199514857
2024-09-27 08:43:07,692 | INFO | All models have been at least evaluated once, t = 10 < 50.
2024-09-27 08:43:07,704 | INFO | Evolving 2 and 9 to 11 and 12
2024-09-27 08:43:12,135 | INFO | Replacing 0 by 10
2024-09-27 08:43:17,496 | INFO | Replacing 1 by 11
2024-09-27 08:43:17,507 | INFO | Evolving 2 and 4 to 13 and 14
2024-09-27 08:43:23,391 | INFO | Replacing 3 by 12
2024-09-27 08:43:23,392 | INFO | Best found ! 0.02850212249848394 < 0.04790782292298368
2024-09-27 08:43:27,831 | INFO | Replacing 5 by 13
2024-09-27 08:43:27,838 | INFO | Evolving 4 and 7 to 15 and 16
2024-09-27 08:43:32,387 | INFO | Replacing 6 by 14
2024-09-27 08:43:35,771 | INFO | Replacing 8 by 15
2024-09-27 08:43:35,778 | INFO | Evolving 2 and 12 to 17 and 18
2024-09-27 08:43:41,317 | INFO | Replacing 10 by 16
2024-09-27 08:43:41,318 | INFO | Best found ! 0.02577319587628868 < 0.02850212249848394
2024-09-27 08:43:47,271 | INFO | Replacing 9 by 17
2024-09-27 08:43:47,272 | INFO | Best found ! 0.020921770770163772 < 0.02577319587628868
2024-09-27 08:43:47,289 | INFO | Evolving 4 and 13 to 19 and 20
2024-09-27 08:43:54,532 | INFO | Replacing 7 by 18
2024-09-27 08:43:59,097 | INFO | Replacing 15 by 19
2024-09-27 08:43:59,108 | INFO | Evolving 16 and 18 to 21 and 22
2024-09-27 08:44:04,720 | INFO | Replacing 11 by 20
2024-09-27 08:44:04,721 | INFO | Best found ! 0.02031534263189816 < 0.020921770770163772
2024-09-27 08:44:12,168 | INFO | Replacing 4 by 21
2024-09-27 08:44:12,181 | INFO | Evolving 12 and 20 to 23 and 24
2024-09-27 08:44:23,990 | INFO | Replacing 13 by 23
2024-09-27 08:44:23,991 | INFO | Best found ! 0.016373559733171672 < 0.02031534263189816
2024-09-27 08:44:24,004 | INFO | Evolving 17 and 16 to 25 and 26
2024-09-27 08:44:35,805 | INFO | Replacing 14 by 25
2024-09-27 08:44:35,818 | INFO | Evolving 23 and 20 to 27 and 28
2024-09-27 08:44:41,739 | INFO | Replacing 19 by 26
2024-09-27 08:44:41,740 | INFO | Best found ! 0.016070345664038865 < 0.016373559733171672
2024-09-27 08:44:47,223 | INFO | Replacing 18 by 27
2024-09-27 08:44:47,234 | INFO | Evolving 23 and 20 to 29 and 30
2024-09-27 08:44:52,682 | INFO | Replacing 21 by 28
2024-09-27 08:44:52,683 | INFO | Best found ! 0.015463917525773141 < 0.016070345664038865
2024-09-27 08:44:58,133 | INFO | Replacing 2 by 29
2024-09-27 08:44:58,145 | INFO | Evolving 28 and 26 to 31 and 32
2024-09-27 08:45:03,599 | INFO | Replacing 12 by 30
2024-09-27 08:45:03,601 | INFO | Best found ! 0.014554275318374721 < 0.015463917525773141
2024-09-27 08:45:09,495 | INFO | Replacing 16 by 31
2024-09-27 08:45:09,496 | INFO | Best found ! 0.014251061249241914 < 0.014554275318374721
2024-09-27 08:45:09,506 | INFO | Evolving 30 and 31 to 33 and 34
2024-09-27 08:45:14,990 | INFO | Replacing 25 by 32
2024-09-27 08:45:20,818 | INFO | Replacing 17 by 33
2024-09-27 08:45:20,819 | INFO | Best found ! 0.0136446331109763 < 0.014251061249241914
2024-09-27 08:45:20,831 | INFO | Evolving 33 and 31 to 35 and 36
2024-09-27 08:45:26,682 | INFO | Replacing 20 by 34
2024-09-27 08:45:26,683 | INFO | Best found ! 0.013341419041843494 < 0.0136446331109763
2024-09-27 08:45:32,536 | INFO | Replacing 27 by 35
2024-09-27 08:45:32,548 | INFO | Evolving 26 and 34 to 37 and 38
2024-09-27 08:45:45,163 | INFO | Replacing 29 by 37
2024-09-27 08:45:45,175 | INFO | Evolving 33 and 35 to 39 and 40
2024-09-27 08:45:54,907 | INFO | Replacing 23 by 39
2024-09-27 08:45:54,921 | INFO | Evolving 33 and 32 to 41 and 42
2024-09-27 08:46:05,588 | INFO | Replacing 26 by 41
2024-09-27 08:46:05,602 | INFO | Evolving 37 and 34 to 43 and 44
2024-09-27 08:46:12,259 | INFO | Replacing 28 by 42
2024-09-27 08:46:18,191 | INFO | Replacing 30 by 43
2024-09-27 08:46:18,192 | INFO | Best found ! 0.01273499090357788 < 0.013341419041843494
2024-09-27 08:46:18,203 | INFO | Evolving 37 and 33 to 45 and 46
2024-09-27 08:46:24,142 | INFO | Replacing 35 by 44
2024-09-27 08:46:30,096 | INFO | Replacing 39 by 45
2024-09-27 08:46:30,108 | INFO | Evolving 43 and 42 to 47 and 48
2024-09-27 08:46:36,131 | INFO | Replacing 31 by 46
2024-09-27 08:46:36,132 | INFO | Best found ! 0.012128562765312267 < 0.01273499090357788
2024-09-27 08:46:42,750 | INFO | Replacing 32 by 47
2024-09-27 08:46:42,761 | INFO | Evolving 47 and 34 to 49 and 50
2024-09-27 08:46:49,455 | INFO | Replacing 41 by 48
2024-09-27 08:46:49,456 | INFO | Best found ! 0.009399636143117007 < 0.012128562765312267

Draw Graph#

After the optimization, it is possible to load the best graph and draw the architecture.

[12]:

import graphviz
from dragon.utils.plot_functions import draw_cell, load_archi, str_operations

def draw_graph(n_dag, m_dag, output_file, act="Identity()", name="Input"):
    G = graphviz.Digraph(output_file, format='pdf',
                            node_attr={'nodesep': '0.02', 'shape': 'box', 'rankstep': '0.02', 'fontsize': '20', "fontname": "sans-serif"})

    G, g_nodes = draw_cell(G, n_dag, m_dag, "#ffa600", [], name_input=name,
                            color_input="#ef5675")
    G.node(','.join(["MLP", "10", act]), style="rounded,filled", color="black", fillcolor="#ef5675", fontcolor="#ECECEC")
    G.edge(g_nodes[-1], ','.join(["MLP", "10", act]))
    return G

best_model = load_archi("save/test/best_model/x.pkl")
labels = [e.label for e in search_space]
best_model = dict(zip(labels, best_model))
m_dag = best_model['Dag'].matrix
n_dag = str_operations(best_model["Dag"].operations)

graph = draw_graph(n_dag, m_dag, "save/test/best_archi")
print(f'Model giving a score of ', np.round((1-min_loss)*100, 2), '%:')
graph

Model giving a score of  99.06 %:

[12]:

Baseline: SVC / RandomForest with Optuna#

[10]:

import sklearn
import optuna
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

def objective(trial):

    classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
    if classifier_name == 'SVC':
         svc_c = trial.suggest_float('svc_c', 1e-10, 1e10, log=True)
         classifier_obj = SVC(C=svc_c, gamma='auto')
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32, log=True)
        classifier_obj = RandomForestClassifier(max_depth=rf_max_depth, n_estimators=10)
    classifier_obj.fit(X_train, y_train, )
    pred = classifier_obj.predict(X_val)
    accuracy = accuracy_score(y_val, pred)
    return accuracy

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"\nBest Accuracy: ", np.round(study.best_value,2)*100)

[I 2024-09-30 08:31:47,051] A new study created in memory with name: no-name-56c0c993-5fe9-4734-b454-5c7296a0f970
[I 2024-09-30 08:31:47,147] Trial 0 finished with value: 0.9802910855063675 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 9}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:47,192] Trial 1 finished with value: 0.7947240751970891 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 3}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:53,258] Trial 2 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 107434903.85230298}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:59,078] Trial 3 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 76.05790764453474}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:04,722] Trial 4 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1278143.6692545111}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:10,627] Trial 5 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 4396269782.010601}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,738] Trial 6 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00011304122770204807}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,813] Trial 7 finished with value: 0.9642207398423287 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 7}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,849] Trial 8 finished with value: 0.7404487568223166 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 2}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:21,528] Trial 9 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1432989.5573470553}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:21,632] Trial 10 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 10 with value: 0.9866585809581565.
[I 2024-09-30 08:32:21,745] Trial 11 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:21,857] Trial 12 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:21,952] Trial 13 finished with value: 0.9851425106124925 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:22,047] Trial 14 finished with value: 0.9848392965433597 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:22,150] Trial 15 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,253] Trial 16 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,357] Trial 17 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,461] Trial 18 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,517] Trial 19 finished with value: 0.8750758035172832 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 4}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,627] Trial 20 finished with value: 0.9854457246816253 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,740] Trial 21 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,840] Trial 22 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,939] Trial 23 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,033] Trial 24 finished with value: 0.9796846573681018 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 10}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,131] Trial 25 finished with value: 0.9845360824742269 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,197] Trial 26 finished with value: 0.9408732565191025 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 6}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,292] Trial 27 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,381] Trial 28 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 13}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,468] Trial 29 finished with value: 0.9881746513038205 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,552] Trial 30 finished with value: 0.9839296543359611 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 10}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,641] Trial 31 finished with value: 0.9896907216494846 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 31 with value: 0.9896907216494846.
[I 2024-09-30 08:32:23,733] Trial 32 finished with value: 0.990600363856883 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:23,826] Trial 33 finished with value: 0.9845360824742269 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 15}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:23,917] Trial 34 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:28,852] Trial 35 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 6.511115455954505e-10}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:28,920] Trial 36 finished with value: 0.957550030321407 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 7}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:33,812] Trial 37 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00035066537122351867}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:33,893] Trial 38 finished with value: 0.9796846573681018 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 9}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,367] Trial 39 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 3.267386529006129e-09}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,460] Trial 40 finished with value: 0.9860521528198909 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,548] Trial 41 finished with value: 0.989084293511219 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,646] Trial 42 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,762] Trial 43 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,878] Trial 44 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,818] Trial 45 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 84.19317057488534}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,909] Trial 46 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,944] Trial 47 finished with value: 0.7195269860521528 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 2}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:47,030] Trial 48 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:47,123] Trial 49 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 27}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,568] Trial 50 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00049795530410561}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,660] Trial 51 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,753] Trial 52 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,847] Trial 53 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,947] Trial 54 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,048] Trial 55 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,148] Trial 56 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,244] Trial 57 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 31}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,336] Trial 58 finished with value: 0.989084293511219 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,429] Trial 59 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,478] Trial 60 finished with value: 0.8641600970285022 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 4}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,575] Trial 61 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,671] Trial 62 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,766] Trial 63 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,863] Trial 64 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,960] Trial 65 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,054] Trial 66 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,140] Trial 67 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 15}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,224] Trial 68 finished with value: 0.9860521528198909 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,763] Trial 69 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 2.4446970641424645e-07}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,851] Trial 70 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,940] Trial 71 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 27}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,036] Trial 72 finished with value: 0.9833232261976955 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,134] Trial 73 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,223] Trial 74 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,311] Trial 75 finished with value: 0.9839296543359611 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,397] Trial 76 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,483] Trial 77 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,568] Trial 78 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:06,825] Trial 79 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1.7260909483357085}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:06,919] Trial 80 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,018] Trial 81 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,110] Trial 82 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,207] Trial 83 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,301] Trial 84 finished with value: 0.9899939357186174 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,391] Trial 85 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,445] Trial 86 finished with value: 0.9129775621588841 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 5}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,534] Trial 87 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,620] Trial 88 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 11}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,700] Trial 89 finished with value: 0.9721043056397817 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 8}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,792] Trial 90 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,886] Trial 91 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,977] Trial 92 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,075] Trial 93 finished with value: 0.9881746513038205 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,173] Trial 94 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,267] Trial 95 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,368] Trial 96 finished with value: 0.9851425106124925 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,144] Trial 97 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 33766.72119670471}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,233] Trial 98 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,320] Trial 99 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.


Best Accuracy:  99.0

Quickstart

Contents