Quickstart#

[1]:

import warnings
warnings.filterwarnings('ignore')
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import openml
from sklearn.model_selection import train_test_split
import numpy as np
import os

Loading the dataset#

This tutorial introduces you to a complete machine learning pipeline to run DRAGON for a simple classification task on the PenDigits dataset.

[2]:

dataset = openml.datasets.get_dataset(32)
data, _, numerical, names = dataset.get_data()
data.head()

[2]:

	input1	input2	input3	input4	input5	input6	input7	input8	input9	input10	input11	input12	input13	input14	input15	input16	class
0	47.0	100.0	27.0	81.0	57.0	37.0	26.0	0.0	0.0	23.0	56.0	53.0	100.0	90.0	40.0	98.0	8
1	0.0	89.0	27.0	100.0	42.0	75.0	29.0	45.0	15.0	15.0	37.0	0.0	69.0	2.0	100.0	6.0	2
2	0.0	57.0	31.0	68.0	72.0	90.0	100.0	100.0	76.0	75.0	50.0	51.0	28.0	25.0	16.0	0.0	1
3	0.0	100.0	7.0	92.0	5.0	68.0	19.0	45.0	86.0	34.0	100.0	45.0	74.0	23.0	67.0	0.0	4
4	0.0	67.0	49.0	83.0	100.0	100.0	81.0	80.0	60.0	60.0	40.0	40.0	33.0	20.0	47.0	0.0	1

[3]:

X = data.drop('class', axis=1)
y = data[["class"]].astype(int)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0)
print(f"X_train: {X_train.shape}, y_train: {y_train.shape}, X_val: {X_val.shape}, y_val: {y_val.shape}")
print(f"Number of features: {X.shape[1]}, number of classes: {y.shape[1]}")

X_train: (7694, 16), y_train: (7694, 1), X_val: (3298, 16), y_val: (3298, 1)
Number of features: 16, number of classes: 1

[4]:

class CustomDataset(Dataset):
    def __init__(self, X, y):
        super().__init__()
        self.X = torch.FloatTensor(X.values)
        self.y = torch.LongTensor(y.values)
    def __len__(self):
        return self.X.shape[0]
    def __getitem__(self, index):
        return self.X[index], self.y[index]
train_set = CustomDataset(X_train, y_train)
val_set = CustomDataset(X_val, y_val)
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
val_loader = DataLoader(val_set, batch_size=1, shuffle=False)

Search Space#

In DRAGON the neural networks are represented as Directed Acyclic Graphs (DAGs), where each node represents nn.Module layer, and the edges are the connection between them. Within the package, the DAGs are encoded using their adjaceny matrix. To know more about these objects, refer to the Search Space section.

Meta-Architecture#

The meta-architecture $f$ should encompasses all the considered neural networks from the search space. To define a meta-architecture usable with DRAGON, we create a class that inherits from nn.Module. The __init__ function should take as input arguments defining a certain set of architecture and hyperparameters $\{\alpha, \theta\}$ . Then, the __forward__ function should take an input a variable $X$ and returns the prediction $\hat{y} = f_{\alpha,\theta}(X)$ .

The main bricks to build neural networks within the DRAGON package are dragon.search_space.cells.AdjMatrix and dragon.search_space.cells.Node. The AdjMatrix creates a DAG and the Node a simple layer. Both bricks can be used to create models adapted to the task at hand. In this case we stay very simple with a single AdjMatrix embodying the neural network and a final Node to get the right output shape.

Before the first training of a given neural network, the layers should be initialized with the right input shape.

[5]:

from dragon.search_space.dag_encoding import AdjMatrix, Node

class MetaArchi(nn.Module):
    def __init__(self, args, input_shape):
        super().__init__()
        # Number of features, here equals to 16
        self.input_shape = input_shape

        # We create the DAG using the WeightsAdjCell module
        assert isinstance(args['Dag'], AdjMatrix), f"The 'Dag' argument should be an 'AdjMatrix'. Got {type(args['Dag'])} instead."
        self.dag = args['Dag']
        self.dag.set(input_shape)

        # We set the final layer
        assert isinstance(args['Out'], Node), f"The 'Out' argument should be a 'Node'. Got {type(args['Node'])} instead."
        self.output = args["Out"]
        self.output.set(self.dag.output_shape)

    def forward(self, X):
        out = self.dag(X)
        return self.output(out)

    def save(self, path):
        if not os.path.exists(path):
            os.makedirs(path)
        full_path = os.path.join(path, "best_model.pth")
        torch.save(self.state_dict(), full_path)

Arguments#

The MetaArchi class takes as input a dictionnary args containing one search space configuration ${\alpha, \theta}$ . We now need to define what values $\alpha$ and $\theta$ can take. To keep it simple, we will only use nn.Linear and nn.Identity layers as candidate operations. Several ready-to-use candidate operations are available within dragon.search_space.bricks_variables. They are then used to build Directed Acyclic Graphs.

[6]:

from dragon.search_space.bricks_variables import mlp_var, identity_var, operations_var, mlp_const_var, dag_var, node_var

candidate_operations = operations_var("Candidate operations", size=10, candidates=[mlp_var("MLP"), identity_var("Identity")])
dag = dag_var("Dag", candidate_operations)
print(f'An example of a generated DAG: {dag.random()}')

# For the last layer, we do not want the number of output channels to be optimized. It should be equal to the number of classes.
out = node_var("Out", operation=mlp_const_var('Operation', 10), activation_function=nn.Softmax())

An example of a generated DAG: NODES: [
(combiner) add -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) Identity() -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 498} -- (activation) LeakyReLU(negative_slope=0.01) -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.MLP'> -- (hp) {'out_channels': 146} -- (activation) LeakyReLU(negative_slope=0.01) -- ,
(combiner) add -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) SiLU() -- ,
(combiner) mul -- (name) <class 'dragon.search_space.bricks.basics.Identity'> -- (hp) {} -- (activation) Sigmoid() -- ] | MATRIX:[[0, 1, 1, 0, 0], [0, 0, 1, 1, 0], [0, 0, 0, 0, 1], [0, 0, 0, 0, 1], [0, 0, 0, 0, 0]]

With the elements dag and out we can generate composants to build a neural network.

[7]:

args = {"Dag": dag.random(), "Out": out.random()}
model = MetaArchi(args, input_shape=(16,))

The final search space is an array which contain all the structural elements of the architecture as well as eventually other elements for the training for example.

[8]:

from dragon.search_space.base_variables import ArrayVar
from dragon.search_operators.base_neighborhoods import ArrayInterval

search_space = ArrayVar(dag, out, label="Search Space", neighbor=ArrayInterval())

Loss function#

The search algorithm should minimize a certain loss function. Typically, it consists in training the model on a train set and then validate it on a validation set.

[9]:

def train_model(model, data_loader):
    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.05)
    model.train()
    for _ in range(2):
        for X,y in data_loader:
            optimizer.zero_grad()
            y = y.squeeze()
            pred = model(X)
            loss = loss_fn(pred,y)
            loss.backward()
            optimizer.step()
    return model

def test_model(model, data_loader):
    loss_fn = nn.CrossEntropyLoss()
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
      for X, y in data_loader:
          y = y.squeeze(1)
          pred = model(X)
          loss = loss_fn(pred, y).item()
          test_loss += loss
          prediction = pred.argmax(axis=1)
          correct += (prediction == y).sum().item()
    accuracy = correct/ len(data_loader.dataset)
    return accuracy

def loss_function(args, idx, *kwargs):
    labels = [e.label for e in search_space]
    args = dict(zip(labels, args))
    model = MetaArchi(args, input_shape=(16,))
    model = train_model(model, train_loader)
    accuracy = test_model(model, val_loader)
    print(f"Idx = {idx}, loss = {1-accuracy}")
    return 1 - accuracy, model

#loss, model = loss_function(search_space.random())
#print(f'For a given argument, accuracy = ', np.round((1-loss)*100,2), '%')

Search Algorithm#

We can use several search algorithms to optimize the structures from the search space. Here, we take the example of Mutant-UCB.

[11]:

from dragon.search_algorithm.mutant_ucb import Mutant_UCB

search_algorithm = Mutant_UCB(search_space, save_dir="save/test_mutant", T=20, N=5, K=5, E=0.01, evaluation=loss_function)
search_algorithm.run()

2024-11-20 16:12:45,709 | WARNING | Install mpi4py if you want to use the distributed version.
2024-11-20 16:12:45,709 | INFO | save/test_mutant already exists. Deleting it.
2024-11-20 16:12:45,716 | INFO | The whole population has been created (size = 5), 5 have been randomy initialized.
Idx = 0, loss = 0.8956943602183142
2024-11-20 16:12:48,703 | INFO | Best found! 0.8956943602183142 < inf
Idx = 1, loss = 0.8935718617343845
2024-11-20 16:12:51,989 | INFO | Best found! 0.8935718617343845 < 0.8956943602183142
Idx = 2, loss = 0.8432383262583384
2024-11-20 16:12:53,606 | INFO | Best found! 0.8432383262583384 < 0.8935718617343845
Idx = 3, loss = 0.78077622801698
2024-11-20 16:12:59,195 | INFO | Best found! 0.78077622801698 < 0.8432383262583384
Idx = 4, loss = 0.6030927835051547
2024-11-20 16:13:06,703 | INFO | Best found! 0.6030927835051547 < 0.78077622801698
2024-11-20 16:13:06,710 | INFO | All models have been at least evaluated once, t = 5 < 20.
2024-11-20 16:13:06,711 | INFO | After initialisation, it remains 15 iterations.
2024-11-20 16:13:06,712 | INFO | With p = 0.2 = 1 / 5, training 4 instead
Idx = 4, loss = 0.49332929047907825
2024-11-20 16:13:14,332 | INFO | Best found! 0.49332929047907825 < 0.6030927835051547
2024-11-20 16:13:14,340 | INFO | With p = 0.4 = 2 / 5, mutating 4 to 5
Idx = 5, loss = 0.3089751364463311
2024-11-20 16:13:20,223 | INFO | Best found! 0.3089751364463311 < 0.49332929047907825
2024-11-20 16:13:20,229 | INFO | With p = 0.2 = 1 / 5, mutating 5 to 6
Idx = 6, loss = 0.44906003638568825
2024-11-20 16:13:28,854 | INFO | With p = 0.2 = 1 / 5, mutating 5 to 7
Idx = 7, loss = 0.2731958762886598
2024-11-20 16:13:34,956 | INFO | Best found! 0.2731958762886598 < 0.3089751364463311
2024-11-20 16:13:34,962 | INFO | With p = 0.2 = 1 / 5, training 7 instead
Idx = 7, loss = 0.11855670103092786
2024-11-20 16:13:41,370 | INFO | Best found! 0.11855670103092786 < 0.2731958762886598
2024-11-20 16:13:41,377 | INFO | With p = 0.4 = 2 / 5, training 7 instead
Idx = 7, loss = 0.06124924196482717
2024-11-20 16:13:48,104 | INFO | Best found! 0.06124924196482717 < 0.11855670103092786
2024-11-20 16:13:48,110 | INFO | With p = 0.6 = 3 / 5, training 7 instead
Idx = 7, loss = 0.024560339599757453
2024-11-20 16:13:54,704 | INFO | Best found! 0.024560339599757453 < 0.06124924196482717
2024-11-20 16:13:54,710 | INFO | With p = 0.8 = 4 / 5, mutating 7 to 8
Idx = 8, loss = 0.01849605821710132
2024-11-20 16:14:01,507 | INFO | Best found! 0.01849605821710132 < 0.024560339599757453
2024-11-20 16:14:01,514 | INFO | With p = 0.2 = 1 / 5, training 8 instead
Idx = 8, loss = 0.01273499090357788
2024-11-20 16:14:07,447 | INFO | Best found! 0.01273499090357788 < 0.01849605821710132
2024-11-20 16:14:07,454 | INFO | With p = 0.4 = 2 / 5, training 8 instead
Idx = 8, loss = 0.01273499090357788
2024-11-20 16:14:14,947 | INFO | With p = 0.6 = 3 / 5, training 8 instead
Idx = 8, loss = 0.01091570648878104
2024-11-20 16:14:21,741 | INFO | Best found! 0.01091570648878104 < 0.01273499090357788
2024-11-20 16:14:21,748 | INFO | With p = 0.8 = 4 / 5, mutating 8 to 9
Idx = 9, loss = 0.20285021224984834
2024-11-20 16:14:30,779 | INFO | With p = 0.8 = 4 / 5, mutating 8 to 10
Idx = 10, loss = 0.01182534869617946
2024-11-20 16:14:36,820 | INFO | With p = 0.2 = 1 / 5, training 10 instead
Idx = 10, loss = 0.015463917525773141
2024-11-20 16:14:42,976 | INFO | With p = 0.4 = 2 / 5, mutating 10 to 11
Idx = 11, loss = 0.01091570648878104
2024-11-20 16:14:49,049 | INFO | Search algorithm is done. Min Loss = 0.01091570648878104

Draw Graph#

After the optimization, it is possible to load the best graph and draw the architecture.

[12]:

import graphviz
from dragon.utils.plot_functions import draw_cell, load_archi, str_operations

def draw_graph(n_dag, m_dag, output_file, act="Identity()", name="Input"):
    G = graphviz.Digraph(output_file, format='pdf',
                            node_attr={'nodesep': '0.02', 'shape': 'box', 'rankstep': '0.02', 'fontsize': '20', "fontname": "sans-serif"})

    G, g_nodes = draw_cell(G, n_dag, m_dag, "#ffa600", [], name_input=name,
                            color_input="#ef5675")
    G.node(','.join(["MLP", "10", act]), style="rounded,filled", color="black", fillcolor="#ef5675", fontcolor="#ECECEC")
    G.edge(g_nodes[-1], ','.join(["MLP", "10", act]))
    return G

best_model = load_archi("save/test/best_model/x.pkl")
labels = [e.label for e in search_space]
best_model = dict(zip(labels, best_model))
m_dag = best_model['Dag'].matrix
n_dag = str_operations(best_model["Dag"].operations)

graph = draw_graph(n_dag, m_dag, "save/test/best_archi")
print(f'Model giving a score of ', np.round((1-search_algorithm.min_loss)*100, 2), '%:')
graph

Model giving a score of  98.91 %:

[12]:

../_images/Quickstart_quickstart_20_1.svg

Baseline: SVC / RandomForest with Optuna#

[ ]:

import sklearn
import optuna
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

def objective(trial):

    classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
    if classifier_name == 'SVC':
         svc_c = trial.suggest_float('svc_c', 1e-10, 1e10, log=True)
         classifier_obj = SVC(C=svc_c, gamma='auto')
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32, log=True)
        classifier_obj = RandomForestClassifier(max_depth=rf_max_depth, n_estimators=10)
    classifier_obj.fit(X_train, y_train, )
    pred = classifier_obj.predict(X_val)
    accuracy = accuracy_score(y_val, pred)
    return accuracy

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
print(f"\nBest Accuracy: ", np.round(study.best_value,2)*100)

[I 2024-09-30 08:31:47,051] A new study created in memory with name: no-name-56c0c993-5fe9-4734-b454-5c7296a0f970
[I 2024-09-30 08:31:47,147] Trial 0 finished with value: 0.9802910855063675 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 9}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:47,192] Trial 1 finished with value: 0.7947240751970891 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 3}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:53,258] Trial 2 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 107434903.85230298}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:31:59,078] Trial 3 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 76.05790764453474}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:04,722] Trial 4 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1278143.6692545111}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:10,627] Trial 5 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 4396269782.010601}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,738] Trial 6 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00011304122770204807}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,813] Trial 7 finished with value: 0.9642207398423287 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 7}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:15,849] Trial 8 finished with value: 0.7404487568223166 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 2}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:21,528] Trial 9 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1432989.5573470553}. Best is trial 0 with value: 0.9802910855063675.
[I 2024-09-30 08:32:21,632] Trial 10 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 10 with value: 0.9866585809581565.
[I 2024-09-30 08:32:21,745] Trial 11 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:21,857] Trial 12 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:21,952] Trial 13 finished with value: 0.9851425106124925 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:22,047] Trial 14 finished with value: 0.9848392965433597 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 11 with value: 0.9887810794420862.
[I 2024-09-30 08:32:22,150] Trial 15 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,253] Trial 16 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,357] Trial 17 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,461] Trial 18 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,517] Trial 19 finished with value: 0.8750758035172832 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 4}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,627] Trial 20 finished with value: 0.9854457246816253 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,740] Trial 21 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,840] Trial 22 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:22,939] Trial 23 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,033] Trial 24 finished with value: 0.9796846573681018 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 10}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,131] Trial 25 finished with value: 0.9845360824742269 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,197] Trial 26 finished with value: 0.9408732565191025 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 6}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,292] Trial 27 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,381] Trial 28 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 13}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,468] Trial 29 finished with value: 0.9881746513038205 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,552] Trial 30 finished with value: 0.9839296543359611 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 10}. Best is trial 15 with value: 0.9893875075803518.
[I 2024-09-30 08:32:23,641] Trial 31 finished with value: 0.9896907216494846 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 31 with value: 0.9896907216494846.
[I 2024-09-30 08:32:23,733] Trial 32 finished with value: 0.990600363856883 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:23,826] Trial 33 finished with value: 0.9845360824742269 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 15}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:23,917] Trial 34 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:28,852] Trial 35 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 6.511115455954505e-10}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:28,920] Trial 36 finished with value: 0.957550030321407 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 7}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:33,812] Trial 37 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00035066537122351867}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:33,893] Trial 38 finished with value: 0.9796846573681018 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 9}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,367] Trial 39 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 3.267386529006129e-09}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,460] Trial 40 finished with value: 0.9860521528198909 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,548] Trial 41 finished with value: 0.989084293511219 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,646] Trial 42 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,762] Trial 43 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:39,878] Trial 44 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,818] Trial 45 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 84.19317057488534}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,909] Trial 46 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:46,944] Trial 47 finished with value: 0.7195269860521528 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 2}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:47,030] Trial 48 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:47,123] Trial 49 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 27}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,568] Trial 50 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 0.00049795530410561}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,660] Trial 51 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,753] Trial 52 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,847] Trial 53 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:52,947] Trial 54 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,048] Trial 55 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,148] Trial 56 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,244] Trial 57 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 31}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,336] Trial 58 finished with value: 0.989084293511219 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,429] Trial 59 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 17}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,478] Trial 60 finished with value: 0.8641600970285022 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 4}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,575] Trial 61 finished with value: 0.9857489387507581 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,671] Trial 62 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,766] Trial 63 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,863] Trial 64 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:53,960] Trial 65 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 19}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,054] Trial 66 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,140] Trial 67 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 15}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:54,224] Trial 68 finished with value: 0.9860521528198909 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 12}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,763] Trial 69 finished with value: 0.10278956943602183 and parameters: {'classifier': 'SVC', 'svc_c': 2.4446970641424645e-07}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,851] Trial 70 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:32:59,940] Trial 71 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 27}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,036] Trial 72 finished with value: 0.9833232261976955 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,134] Trial 73 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,223] Trial 74 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 24}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,311] Trial 75 finished with value: 0.9839296543359611 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 14}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,397] Trial 76 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 32}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,483] Trial 77 finished with value: 0.9872650090964221 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:00,568] Trial 78 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:06,825] Trial 79 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 1.7260909483357085}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:06,919] Trial 80 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,018] Trial 81 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,110] Trial 82 finished with value: 0.9893875075803518 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,207] Trial 83 finished with value: 0.9866585809581565 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,301] Trial 84 finished with value: 0.9899939357186174 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,391] Trial 85 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 28}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,445] Trial 86 finished with value: 0.9129775621588841 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 5}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,534] Trial 87 finished with value: 0.9887810794420862 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 21}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,620] Trial 88 finished with value: 0.9869617950272893 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 11}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,700] Trial 89 finished with value: 0.9721043056397817 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 8}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,792] Trial 90 finished with value: 0.9863553668890237 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 16}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,886] Trial 91 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 25}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:07,977] Trial 92 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 29}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,075] Trial 93 finished with value: 0.9881746513038205 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 18}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,173] Trial 94 finished with value: 0.9875682231655549 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 26}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,267] Trial 95 finished with value: 0.9878714372346877 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 22}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:08,368] Trial 96 finished with value: 0.9851425106124925 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 30}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,144] Trial 97 finished with value: 0.11552456033959976 and parameters: {'classifier': 'SVC', 'svc_c': 33766.72119670471}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,233] Trial 98 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 23}. Best is trial 32 with value: 0.990600363856883.
[I 2024-09-30 08:33:14,320] Trial 99 finished with value: 0.9884778653729533 and parameters: {'classifier': 'RandomForest', 'rf_max_depth': 20}. Best is trial 32 with value: 0.990600363856883.


Best Accuracy:  99.0

Quickstart

Contents