{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"## 05. 实现一个简单的AutoML系统"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"### AutoML系统的顶层设计"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们在上个教程中，学习到了UltraOpt的设计哲学是将优化器与评价器分离，如图所示:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from graphviz import Digraph; g = Digraph()\n",
"g.node(\"config space\", shape=\"ellipse\"); g.node(\"optimizer\", shape=\"box\")\n",
"g.node(\"config\", shape=\"ellipse\"); g.node(\"loss\", shape=\"circle\"); g.node(\"evaluator\", shape=\"box\")\n",
"g.edge(\"config space\", \"optimizer\", label=\"initialize\"); g.edge(\"optimizer\", \"config\", label=\"<ask>\", color='blue')\n",
"g.edge(\"config\",\"evaluator\" , label=\"send to\"); g.edge(\"evaluator\",\"loss\" , label=\"evaluate\")\n",
"g.edge(\"config\", \"optimizer\", label=\"<tell>\", color='red'); g.edge(\"loss\", \"optimizer\", label=\"<tell>\", color='red')\n",
"g.graph_attr['rankdir'] = 'LR'; g"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"而当我们要解决AutoML问题时，我们可以这样定义AutoML系统结构："
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"### 用HDL定义一个简单的AutoML配置空间"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"在`03. Conditional Parameter`教程中，我们知道了AutoML问题的优化可以视为一个CASH问题，不仅涉及算法选择，还涉及超参优化。我们用HDL定义一个简单的CASH问题的配置空间："
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from ultraopt.hdl import hdl2cs, plot_hdl, layering_config, plot_layered_dict\n",
"HDL = {\n",
" 'classifier(choice)':{\n",
" \"LinearSVC\": {\n",
" \"max_iter\": {\"_type\": \"int_quniform\",\"_value\": [300, 3000, 100], \"_default\": 600},\n",
" \"penalty\": {\"_type\": \"choice\", \"_value\": [\"l1\", \"l2\"],\"_default\": \"l2\"},\n",
" \"dual\": {\"_type\": \"choice\", \"_value\": [True, False],\"_default\": False},\n",
" \"loss\": {\"_type\": \"choice\", \"_value\": [\"hinge\", \"squared_hinge\"],\"_default\": \"squared_hinge\"},\n",
" \"C\": {\"_type\": \"loguniform\", \"_value\": [0.01, 10000],\"_default\": 1.0},\n",
" \"multi_class\": \"ovr\",\n",
" \"random_state\": 42,\n",
" \"__forbidden\": [\n",
" {\"penalty\": \"l1\",\"loss\": \"hinge\"},\n",
" {\"penalty\": \"l2\",\"dual\": False,\"loss\": \"hinge\"},\n",
" {\"penalty\": \"l1\",\"dual\": False},\n",
" {\"penalty\": \"l1\",\"dual\": True,\"loss\": \"squared_hinge\"},\n",
" ]\n",
" },\n",
" \"RandomForestClassifier\": {\n",
" \"n_estimators\": {\"_type\": \"int_quniform\",\"_value\": [10, 200, 10], \"_default\": 100},\n",
" \"criterion\": {\"_type\": \"choice\",\"_value\": [\"gini\", \"entropy\"],\"_default\": \"gini\"},\n",
" \"max_features\": {\"_type\": \"choice\",\"_value\": [\"sqrt\",\"log2\"],\"_default\": \"sqrt\"},\n",
" \"min_samples_split\": {\"_type\": \"int_uniform\", \"_value\": [2, 20],\"_default\": 2},\n",
" \"min_samples_leaf\": {\"_type\": \"int_uniform\", \"_value\": [1, 20],\"_default\": 1},\n",
" \"bootstrap\": {\"_type\": \"choice\",\"_value\": [True, False],\"_default\": True},\n",
" \"random_state\": 42\n",
" },\n",
" \"KNeighborsClassifier\": {\n",
" \"n_neighbors\": {\"_type\": \"int_loguniform\", \"_value\": [1,100],\"_default\": 3},\n",
" \"weights\" : {\"_type\": \"choice\", \"_value\": [\"uniform\", \"distance\"],\"_default\": \"uniform\"},\n",
" \"p\": {\"_type\": \"choice\", \"_value\": [1, 2],\"_default\": 2},\n",
" },\n",
" }\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"CS = hdl2cs(HDL)\n",
"g = plot_hdl(HDL)\n",
"g.graph_attr['size'] = \"15,8\"\n",
"g"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"### 定义一个简单的AutoML评价器"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"在定义好了配置空间之后，现在我们来看评价器。\n",
"\n",
"在UltraOpt的设计哲学中，评价器可以是一个类（实现`__call__`魔法方法），也可以是个函数。但其必须满足：\n",
"\n",
"- 接受 `dict` 类型参数 `config` \n",
"- 返回 `float` 类型参数 `loss` \n",
"\n",
"在AutoML问题中，评价器的工作流程如下：\n",
"1. 将config转化为一个机器学习模型\n",
"2. 在训练集上对机器学习模型进行训练\n",
"3. 在验证集上得到相应的评价指标\n",
"4. 对评价指标进行处理，使其`越小越好`，返回`loss`\n",
"\n",
"在了解了这些知识后，我们来开发 **AutoML 评价器**"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"为了方便同学们理解，我们先顺序地将之前提到的**AutoML 评价器**工作流程跑一遍：\n",
"\n",
"#### Step 1. 将config转化为一个机器学习模型"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"# 引入 sklearn 的分类器\n",
"from sklearn.svm import LinearSVC\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.neighbors import KNeighborsClassifier"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"假设评价器传入了一个 `config` 参数，我们先通过获取配置空间默认值（也可以对配置空间 `CS` 对象进行采样）得到 `config`"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"{'classifier:__choice__': 'LinearSVC',\n",
" 'classifier:LinearSVC:C': 1.0,\n",
" 'classifier:LinearSVC:dual': 'True:bool',\n",
" 'classifier:LinearSVC:loss': 'squared_hinge',\n",
" 'classifier:LinearSVC:max_iter': 600,\n",
" 'classifier:LinearSVC:multi_class': 'ovr',\n",
" 'classifier:LinearSVC:penalty': 'l2',\n",
" 'classifier:LinearSVC:random_state': '42:int'}"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"config = CS.get_default_configuration().get_dictionary()\n",
"config"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们用**配置分层函数** `ultraopt.hdl.layering_config` 处理这个配置"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"{'classifier': {'LinearSVC': {'C': 1.0,\n",
" 'dual': True,\n",
" 'loss': 'squared_hinge',\n",
" 'max_iter': 600,\n",
" 'multi_class': 'ovr',\n",
" 'penalty': 'l2',\n",
" 'random_state': 42}}}"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"layered_dict = layering_config(config)\n",
"layered_dict"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"plot_layered_dict(layered_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们需要获取这个配置的如下信息：\n",
"\n",
"- 算法选择的结果\n",
"- 被选择算法对应的参数"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"'LinearSVC'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"AS_HP = layered_dict['classifier'].copy()\n",
"AS, HP = AS_HP.popitem()\n",
"AS # 算法选择的结果"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"{'C': 1.0,\n",
" 'dual': True,\n",
" 'loss': 'squared_hinge',\n",
" 'max_iter': 600,\n",
" 'multi_class': 'ovr',\n",
" 'penalty': 'l2',\n",
" 'random_state': 42}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"HP # 被选择算法对应的参数"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"根据 **算法选择结果** + **对应的参数** 实例化一个 **机器学习对象**"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
" intercept_scaling=1, loss='squared_hinge', max_iter=600,\n",
" multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
" verbose=0)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ML_model = eval(AS)(**HP)\n",
"ML_model # 实例化的机器学习对象"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"#### Step 2. 在训练集上对机器学习模型进行训练"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们采用MNIST手写数字数据集的一个子集来作为训练数据："
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from sklearn.datasets import load_digits\n",
"import seaborn as sns\n",
"import warnings\n",
"warnings.filterwarnings(\"ignore\")\n",
"X, y = load_digits(return_X_y=True)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAAD8CAYAAABAWd66AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXxU1f3/8ddnskAChLUICVSwKMWvKCgiiKWgFRQBsSCLdcNaEKxF/SlSRG3dV1SqaBEQLIhQF1BAwFIVQamggECUJYCShEXWsJNMPr8/MokBswwhc+/J8Hn6uA9m7iznbebOJyfnnnuvqCrGGGP8F/A7gDHGmDxWkI0xxhFWkI0xxhFWkI0xxhFWkI0xxhFWkI0xxhFWkI0xphgiMl5EtovIqkLr/iYiGSKyPLR0Kea1V4jIGhFZLyLDwmrP5iEbY0zRRKQ9sB94Q1XPCa37G7BfVZ8t4XUxwFrgciAdWAL0U9XUktqzHrIxxhRDVRcAu8rw0tbAelXdoKpHgbeAq0t7UWwZGjohsfEp1gUP+Xv9Dn5H4O4xbf2OAEDWE2/7HYHzV/zodwQAth3Y43cEZ+QczZCTfY/sHRvCrjnxv/jVQGBAoVVjVHVMGC/9s4jcCCwF/p+q7j7u8RRgc6H76cBFpb2p9ZCNMacsVR2jqq0KLeEU41eAXwEtgC3Ac+WVJ+I9ZGOM8VRuMKJvr6rb8m+LyGvAzCKelgE0LHS/QWhdiawgG2OiSzAnom8vIvVVdUvo7jXAqiKetgQ4U0Qak1eI+wLXlfbeVpCNMVFFNbfc3ktEpgAdgDoikg48BHQQkRaAApuAgaHnJgNjVbWLquaIyJ+BuUAMMF5VV5fWnhVkY0x0yS2/gqyq/YpYPa6Y52YCXQrdnw3MPpH2rCAbY6JLOfaQvWYF2RgTXSK8Uy+SrCAbY6KL9ZCNMcYNGuFZFpHk9IEhnTt1YPWqBXyXupCh995+yue4feEL/Gnuk9w6+3Fu+eART9p8aPJ/6Dh8LD2fmFyw7uVZi7n2yTfp/dQUbnt5Otv37vckS76Enj2p/frr1H79dRJ79fK07XzP/eMRVqxdwPzPp/vSfj5Xtk1XcgB5O/XCXRzjbEEOBAKMevExuna7nubndaRPnx40a3bmKZsj36S+jzK2y3DGd3vAk/a6X9SM0YO6H7PupkvP59/DrmPaff1of05jxsxZ4kkWgJjGjUns2pWdt93GzltvJb5tW2JSUjxrP9+0KdP5Q6+BnrdbmCvbpis5Cmhu+ItjSi3IIvJrEblPREaFlvtEpFmkg7W+sCVpaZvYuPEHsrOzmTZtBt27dY50s87m8MsFTVJISqx8zLqqCfEFtw8dyeakTz5wAmJ/+UuyU1PhyBEIBslevpxKv/mNhwny/O/zr9ize6/n7RbmyrbpSo4CucHwF8eUWJBF5D7yzlIkwJehRYAp4Z7fs6ySU+qxOT2z4H56xhaSk+tFskmnc+RRrps0jFtmPkrLfh19ypDnHzO/oPODrzP7qzUM6tLGs3ZzNm4k7txzkaQkqFSJ+DZtiKlb17P2XeLKtulKjgIVuIdc2k69PwL/p6rZhVeKyEhgNfBkUS8SkQGEzqAkMdUJBKqUQ1TzRs+H2bdtN4m1k7hu0jB2pG1h85ff+ZLljq5tuaNrW8bNW8pbn61gsEdFOfjDDxyYMoWazzyDHj5Mzvr1qINjgcZHUbxTLxdILmJ9/dBjRSp8BqWyFuPMjK00bPBT0w1S6pOZubVM73UyXMkBsG9b3hn+Du7MYs3cpSS3OMOXHIV1adWU+SvSPG3z8OzZ7Bo4kN1DhqD79hHcvLn0F0UhV7ZNV3IUiOKdencC80XkQxEZE1rmAPOBIZEMtmTpcpo0aUyjRg2Ji4ujd++r+WDmvEg26XSOuIRKxFepXHD7jPbN+XFNuuc5AL7f/tP5ez9ZuYHGdWt62r7UqAFAoG5dKrVvz+H58z1t3xWubJuu5MinGgx7cU2JQxaqOkdEziLv7Pf5u7IzgCUa4f+bYDDIkDtHMHvWm8QEAkyYOJXU1LWRbNLpHFXqJNFrzF0ABGJjWD3jczZ8+k3E2x02YQ5L12ewZ/9hOj0wnkFdLmJh6vds2r6bgAj1a1bj/j7ejmfXePhhAklJaE4O+154Ad3v7bQ7gJfHPkPbdhdSq3YNlq6az7NPvsxbk971NIMr26YrOQo4ODYcrohfU8+uGPITu2LIT+yKIT+xK4b8pDyuGHL46/fDrjmVz+/u5SShUtmResaY6FKBe8hWkI0x0SWYXfpzHGUF2RgTXRycPREuK8jGmOhiQxbGGOMI6yEbY4wjrCAbY4wb1HbqGWOMI2wM2YTDhYMyYi/oUvqTPFC5sb8ndgfY9PYHfkcA4OkLvDm3dUke2vKJ3xHKjw1ZGGOMI6yHbIwxjrAesjHGOMJ6yMYY44icinuCeivIxpjoYj1kY4xxRDmOIYvIeKArsF1VzwmtewboBhwF0oD+qvqzc6iKyCZgHxAEclS1VWntlXrVaWOMqVDK9yKnE4Arjlv3EXCOqp4LrAX+WsLrO6pqi3CKMVhBNsZEm3K8pp6qLgB2HbdunqrmD1QvBhqUV3QryMaY6FK+PeTS3AJ8WFwSYJ6IfCUiA8J5M6fHkDt36sDIkQ8TEwgw/vUpPP3My6dUjocm/4cFqzdRq1oC7/z1DwC8PGsxn6zcgIhQq2oCD1//O+pWrxqxDCMeH8mCRV9Sq2YNpk96NS/DuEm88/4cataoDsCQgTfR/uLWEcsAkHDLPcSedxGatYf9D/wJAKlSjYRBIwjUOY3cHds4OPoROBi56+u58rMo7PaFL3D0wGE0mEtuMMj4bv4c9efKdxU4oVkWoUJZuFiOUdUxYb72fiAHmFzMUy5R1QwRqQt8JCLfhXrcxXK2IAcCAUa9+BhXdOlHevoWFn8xmw9mzuPbb9edMjm6X9SMvu3PZcSkjwrW3XTp+dx+VRsA3vx0BWPmLGFEBC8y2qPL5VzXszvDH3n2mPU39OlB/+t6Razd4x1dOJcj86eTeOt9BesqdelLMHUZB2e/RaUufal8VV8O/3tsxDK48rM43qS+j3Jot/cXes3nyne1wAlcJzRUfMMqwIWJyM3k7ey7TIu5MKmqZoT+3S4i75F3segSC7KzQxatL2xJWtomNm78gezsbKZNm0H3bp1PqRwXNEkhKbHyMeuqJsQX3D50JJtIX6GxVYvmVE+qFuFWShdcuxLdv++YdbEtL+boorzLzR9dNI/Ylu0imsGVn4VrXPmuFijHMeSiiMgVwFCgu6oeLOY5VUSkWv5toBOwqrT3LnNBFpH+ZX1tOJJT6rE5PbPgfnrGFpKT60WySadzFPaPmV/Q+cHXmf3VGgZ1aeNLhinvfMA1Nw5ixOMj2Zu1r/QXRECgek10b97+Ft27i0D1mr7k8PdnoVw3aRi3zHyUlv0i95dSSZz7jpRjQRaRKcAXQFMRSReRPwIvAdXIG4ZYLiKvhp6bLCKzQy89DVgoIiuAL4FZqjqntPZOpof89xL+JwaIyFIRWZqbe+AkmjBFuaNrW+Y+3J8uFzTlrc9WeN5+n2uu4sNp43lnwsv8onYtnnnpNc8zFOkE/lQtL37/LN7o+TDjrhrBWzc9zQU3Xk7D1r/2tH0nleNOPVXtp6r1VTVOVRuo6jhVbaKqDUPT2Vqo6m2h52aqapfQ7Q2qel5o+T9VfSyc6CUWZBH5pphlJXm/AYr7nxijqq1UtVUgUCWcHD+TmbGVhg2SC+43SKlPZubWMr3XyXAlR1G6tGrK/BVpnrdbp1ZNYmJiCAQC9Op+JatS13qeASB3726kei0ApHotcrN+Njc/4vz+WezbthuAgzuzWDN3KcktzvC0fXDwOxIMhr84prQe8mnAjeQdlXL8sjOSwZYsXU6TJo1p1KghcXFx9O59NR/MnBfJJp3Oke/77T8VnU9WbqBxXe//TP9xx0/TMud/+jlNzjjd8wwAOcu/IL5dJwDi23UiZ9nnnmfw82cRl1CJ+CqVC26f0b45P65J96z9fK59RyI9hhxJpc2ymAlUVdXlxz8gIp9EJFFIMBhkyJ0jmD3rTWICASZMnEqqDz0xP3MMmzCHpesz2LP/MJ0eGM+gLhexMPV7Nm3fTUCE+jWrcX8EZ1gA3PvQkyxZ9g179mRxWY/rGfzHG1iy7BvWrNsAAin1TuOhoX+JaAaAhIHDif31eUjV6lR7bgqHp0/kyKy3SBw8grj2V6A7tnPwlUcimsGVn0W+KnWS6DXmLgACsTGsnvE5Gz79xrP287nyXS3gYKENlxQzY6PcxManeD+w56h9H5R0hKU3XLliyMF7w5onH1GJz5zwbKeIsCuG/CTnaMZJTxw6NPbusGtOwq0jIz1R6YQ4Ow/ZGGPKQnMrbh/QCrIxJrpU4CELK8jGmOji4OyJcFlBNsZEF+shG2OMI6wgG2OMI3w4YrO8WEE2xkQX6yEbY4wjbNqb266s19LvCIAbB2W0bX6T3xEAWL5zg98R2Hv6/X5HAOD38Xv9jsBDfgcoTzbLwhhj3KA2ZGGMMY6wIQtjjHFE+Vy81BdWkI0x0cV6yMYY44gc26lnjDFusCELY4xxhA1ZGGOMG2zamzHGuKIC95BLu8iprzp36sDqVQv4LnUhQ++93ZcMcZXieO79kYya8w9e/s/LXHf3dZ61PeLxkbS/qi89rr+tYN3L4yZx6dXX0/Om2+l50+0s+PxLz/KcllyXV99+kWmf/oupn7xB31t7edZ2YX5tF/FXDyTx3ldJGPx0wbqYsy8iYfAzJD40mUCy91d8rnVzD86Y/QqNZ40m+fmhSHyc5xnAje9qgVwNf3GMsz3kQCDAqBcf44ou/UhP38LiL2bzwcx5fPvtOk9zZB/J5v6+wzl88DAxsTE89c7TfPXxV6xZtibibffocjnX9ezO8EeePWb9DX160P8674thTk6Q5//+MmtWriWxSgL/mjuO/y1Yysa1mzzL4Od2kbP8U3K+nEulawYXrMvdvpnDU0dSqdutEW//eLGn1abmjd3ZcOVt6JGjpLz4V5K6/pa97/7H0xyufFcLVOBDp53tIbe+sCVpaZvYuPEHsrOzmTZtBt27dfYly+GDhwGIjY0lNjaGSF8YNl+rFs2pnlTNk7bCsXP7TtaszLua8MEDh9i0bhN169XxNIOf20Xu99+hh/Yfs053ZKI7t3jSflEkNgapHA8xASShEjnbd3qewaXvKuRdUy/cxTWlFmQR+bWIXCYiVY9bf0XkYkFySj02p2cW3E/P2EJycr1INlmsQCDAix+O4l/LJrFs4XLWLvfxEufAlHc+4JobBzHi8ZHszdrnS4b6DerRtPlZrPo61dN2Xdou/JazbSc7x73LmZ9O5MzPJ5O77wAHFi7zPIdzn0kFHrIosSCLyF+AGcAdwCoRubrQw4+X8LoBIrJURJbm5h4on6Q+ys3NZciVf6H/RTdz1nln8cuzTvctS59rruLDaeN5Z8LL/KJ2LZ556TXPMyQkJvD0uEd57sFRHNh/0PP2TZ5AUlWqXdaG9Zf2Z1276wkkVCape0e/Y/kvNzf8xTGl9ZD/BFygqj2ADsADIjIk9JgU9yJVHaOqrVS1VSBQpUzBMjO20rBBcsH9Bin1yczcWqb3Ki8Hsg6w8otvuKDD+b5lqFOrJjExMQQCAXp1v5JVqd721mNiY3h63KPMefcjPp69wNO2wc3twi9VLm5BdvpWgruyICfIvnmLSDy/mec5nPtMorWHDARUdT+Aqm4iryhfKSIjKaEgl4clS5fTpEljGjVqSFxcHL17X80HM+dFsskiJdVKokpS3i+V+ErxtPhNS9LT0j3Pke/HHbsKbs//9HOanOFtb/3BkcPYuG4Tk/851dN287myXbgge8uPJLT4NVK5EgCJbVtwJG2z5zmc+0zKsSCLyHgR2S4iqwqtqyUiH4nIutC/NYt57U2h56wTkbBORF7aLIttItJCVZcDqOp+EekKjAeah9NAWQWDQYbcOYLZs94kJhBgwsSppHrcGwSoVbcWd468i0BMgEAgwMKZn7Fk/hJP2r73oSdZsuwb9uzJ4rIe1zP4jzewZNk3rFm3AQRS6p3GQ0P/4kkWgPNaN+eqa69gXWoakz8aD8DoJ8aw6L+LPcvg53ZRqecdBBo1QxKrkXD3S2R//DZ6aD/xXW5GEpOofN1Qgls3cWTSk57kObxiDVlzFtJ4+ig0GORI6gb2TP3Qk7YLc+W7mk+D5ToUMQF4CXij0LphwHxVfVJEhoXu31f4RSJSi7zz/rcCFPhKRN5X1d0lNSYlzRgQkQZAjqr+7O8PEWmnqotK+7+JjU/x/e8CV64Y8u7Xo/yOYFcMKWTv8PZ+RwDgh8n+XzGk+fcr/I4AQM7RjJP+yzvrj5eHXXOSxn1Uansi0giYqarnhO6vATqo6hYRqQ98oqpNj3tNv9BzBobu/zP0vCkltVViD1lVi/3bPJxibIwxXjuR6WwiMgAYUGjVGFUdU8rLTlPV/LmOW4HTinhOClB4/Cg9tK5Ezh4YYowxZXICBTlUfEsrwCW9XkWk3EYBnD0wxBhjyiT3BJay2RYaqiD07/YinpMBNCx0v0FoXYmsIBtjoorm5Ia9lNH7QP7OmJvIO1bjeHOBTiJSMzQLo1NoXYmsIBtjoks59pBFZArwBdBURNJF5I/Ak8DlIrIO+F3oPiLSSkTGAqjqLuARYEloeTi0rkQ2hmyMiSrleY4KVe1XzEOXFfHcpcCthe6PJ2+KcNisIBtjoot7R0SHzQqyMSaquHgWt3CdEgW5QSDR7wgA5MwY7XcEJw7IcIULB2SYCLAesjHGuEFz/E5QdlaQjTFRRa2HbIwxjrCCbIwxbrAesjHGOMIKsjHGOEKDEb12RkRZQTbGRBXrIRtjjCM013rIxhjjhIrcQ3b6bG+dO3Vg9aoFfJe6kKH33u5rFgkIw2c9xeBx95X+5HLy0IfL6fjSXHqO/6Rg3ciPU+kx9r9c+/on3PXeErIOZ3uWB9z4TFzIAFDr5h6cMfsVGs8aTfLzQ5H4uFMyA7jzmQCoStiLa5wtyIFAgFEvPkbXbtfT/LyO9OnTg2bNzvQtz6X9u7B1fannly5X3c9pyOheFx2zrk2jOrx9Swf+3b8Dp9eswvjF6zzL48Jn4kIGgNjTalPzxu5svGYIG68ajARiSOr621MuA7jzmeTT3PAX1zhbkFtf2JK0tE1s3PgD2dnZTJs2g+7dOvuSpUa9Wpxz6fksemu+p+1e0LA2SQnxx6y7uHFdYgN5H9u5yTXZtu+wZ3lc+ExcyJBPYmOQyvEQE0ASKpGzfecpmcGlzwQgNyhhL64ptSCLSGsRuTB0+2wRuVtEukQ6WHJKPTanZxbcT8/YQnJyvUg3W6RrH7yZ956YRG4JV+j2w/SVm7nkjLqetefCZ+JCBoCcbTvZOe5dzvx0Imd+PpncfQc4sHDZKZcB3PlM8mmuhL24psSCLCIPAaOAV0TkCeAloAowTETuL+F1A0RkqYgszc09UK6BvXbOpeezb+defli10e8ox3jti7XEBIQuZ5d6IVsTAYGkqlS7rA3rL+3PunbXE0ioTFL3jqdcBhdV5IJc2iyLXkALoBJ5l7tuoKpZIvIs8D/gsaJeVPhKrrHxKWXqVmZmbKVhg+SC+w1S6pOZubUsb3VSftWqKef+rhXndGxJbKV4EqomcPPzdzDhrn94niXfjJWb+SxtO//s0wYR7zYqFz4TFzIAVLm4BdnpWwnuygJg37xFJJ7fjKz3Pz6lMoA7n0k+x/6QPSGlDVnkqGpQVQ8CaaqaBaCqh4jwKTyWLF1OkyaNadSoIXFxcfTufTUfzJwXySaLNOPpKQxvO4gRl/yZcXe8wJrPV/lajBdt2M7EL9fzwu8vJCHO21mLLnwmLmQAyN7yIwktfo1UrgRAYtsWHEnbfMplAHc+k3zR3EM+KiKJoYJ8Qf5KEalOhAtyMBhkyJ0jmD3rTWICASZMnEpq6tpINumcYe9/xdLNO9lz6CidRn/EoEuaMn7xOo4Gc7lt2mIAzq1fkxGdz/UkjwufiQsZAA6vWEPWnIU0nj4KDQY5krqBPVM/POUygDufST4Xp7OFS7SE/r2IVFLVI0WsrwPUV9WVpTVQ1iGL8vSn5HZ+RwBg5AO/9DsC1QZN8TuCM1aefp7fEZzR/PsVfkcAIOdoxklX07XNrgi75pz17RynqneJPeSiinFo/Q5gR0QSGWPMSajIPWQ7dNoYE1VcHBsOlxVkY0xUqcizLKwgG2OiivWQjTHGEcFcZ88IUSoryMaYqFKRhywq7q8SY4wpQq5K2EtJRKSpiCwvtGSJyJ3HPaeDiOwt9JwHTya79ZCNMVGlvKa9qeoa8k4dgYjEABnAe0U89TNV7VoebVpBNsZElQgNWVxG3ukjvo/Iu4ecEgW5gfpzFYXjbXoy1e8IppBaKQf9jgDAroxEvyNEldKGIgoTkQHAgEKrxoROjna8vkBxh7m2FZEVQCZwj6quDjvAcU6JgmyMOXWcyCyLwmemLI6IxAPdgb8W8fDXwOmquj90nvjpQJkvl2I79YwxUUVPYAnTlcDXqrrtZ22pZqnq/tDt2UBc6Fw/ZWI9ZGNMVDmRIYsw9aOY4QoRqQdsU1UVkdbkdXLLfB0tK8jGmKhSnicXEpEqwOXAwELrbstrR18l7yIeg0QkBzgE9NWSTqFZCivIxpioUp4nalfVA0Dt49a9Wuj2S+Rd2q5cWEE2xkQVxc5lYYwxTsix8yEbY4wbrIdsjDGOiOjFPiPM6XnInTt1YPWqBXyXupCh997uW47bF77An+Y+ya2zH+eWDx7xJUOtm3twxuxXaDxrNMnPD0Xi/Tn60IXPxIUMAAk9e1L79dep/frrJPbq5UsG2y5+TpGwF9c4W5ADgQCjXnyMrt2up/l5HenTpwfNmpX5AJiTNqnvo4ztMpzx3R7wvO3Y02pT88bubLxmCBuvGowEYkjq+lvPc7jwmbiQASCmcWMSu3Zl5223sfPWW4lv25aYlBRPM9h2UbTcE1hcc8IFWUTeiESQ47W+sCVpaZvYuPEHsrOzmTZtBt27dfaiaSdJbAxSOR5iAkhCJXK2l3nueZm58Jm4kAEg9pe/JDs1FY4cgWCQ7OXLqfSb33iew7aLnwsiYS+uKXEMWUTeP34V0FFEagCoavdIBUtOqcfm9MyC++kZW2h9YctINVcK5bpJw1CFZZPns2zKx562nrNtJzvHvcuZn04k98hRDiz8mgMLl3maAdz4TFzIAJCzcSNVb70VSUpCjxwhvk0bctas8TaDbRdFqsBXcCp1p14DIBUYS96h3wK0Ap4r6UWFz6AkMdUJBKqcfFIfvdHzYfZt201i7SSumzSMHWlb2Pzld561H0iqSrXL2rD+0v4Esw7QYNRwkrp3JOt9b38xmJ8Ef/iBA1OmUPOZZ9DDh8lZvx7N9faPYNsuipbrYM83XKUNWbQCvgLuB/aq6ifAIVX9VFU/Le5FqjpGVVupaquyFuPMjK00bJBccL9BSn0yM7eW6b1O1r5tuwE4uDOLNXOXktziDE/br3JxC7LTtxLclQU5QfbNW0Ti+c08zQBufCYuZMh3ePZsdg0cyO4hQ9B9+whu3uxp+7ZdFC0CJxfyTIkFWVVzVfV5oD9wv4i8hEdT5ZYsXU6TJo1p1KghcXFx9O59NR/MnOdF08eIS6hEfJXKBbfPaN+cH9eke5ohe8uPJLT4NVK5EgCJbVtwJM3bLz+48Zm4kCGf1KgBQKBuXSq1b8/h+fM9bd+2i6JV5J16YRVXVU0HrhWRq4CsyEbKEwwGGXLnCGbPepOYQIAJE6eSmrrWi6aPUaVOEr3G3AVAIDaG1TM+Z8On33ia4fCKNWTNWUjj6aPQYJAjqRvYM/VDTzOAG5+JCxny1Xj4YQJJSWhODvteeAHdv9/T9m27KFquVNwhCzmJExOFJTY+xfe/DP5ev4PfEQD4ffxuvyPQ/PsVfkdwRsbF/k3NKsyFK4a4sl3kHM046Wo6tf4fwq45fbZMdqp625F6xpioEs2zLIwxpkKpyLMsrCAbY6KK72OkJ8EKsjEmqtiQhTHGOMLF6WzhsoJsjIkqQeshG2OMG6yHbIwxjrCC7Lh0yfY7AgC//EN1vyPA434HyHNalRp+RyDphlZ+RwBg9fAf/I4QVSrwJfVOjYJsjDl1WA/ZGGMcEfQ7wEmwgmyMiSo2D9kYYxxhQxbGGOOIilyQnb3qtDHGlEV5XjFERDaJyEoRWS4iS4t4XERklIisF5FvROT8k8luPWRjTFSJwBhyR1XdUcxjVwJnhpaLgFdC/5aJFWRjTFTxeJbF1cAbmnelj8UiUkNE6qvqlrK8mQ1ZGGOiSi4a9iIiA0RkaaFlwHFvp8A8EfmqiMcAUoDCFzJMD60rE6cLcudOHVi9agHfpS5k6L23+5pFAsLwWU8xeNx9nrUZf/VAEu99lYTBTxesizn7IhIGP0PiQ5MJJHt79Wtw4zN57h+PsGLtAuZ/Pt3Tdh/6cDkdX5pLz/GfFKwb+XEqPcb+l2tf/4S73ltC1mHvjgpN/FV9Lpz/dMHSfv0EGgzo4ln7hbmwXeQ7kYucquoYVW1VaBlz3Ntdoqrnkzc0cbuItI9kdmcLciAQYNSLj9G12/U0P68jffr0oFkz/66Bdmn/Lmxdn+FpmznLP+XwpCePWZe7fTOHp44k9/vvPM0C7nwm06ZM5w+9BnrebvdzGjK617HDg20a1eHtWzrw7/4dOL1mFcYvXudZnoNpW1hy2dC85fL7CB46yo7ZX3rWfj5Xtot85blTT1UzQv9uB94DWh/3lAygYaH7DULryuSECrKIXCIid4tIp7I2GK7WF7YkLW0TGzf+QHZ2NtOmzaB7t86RbrZINerV4pxLz2fRW95e5j33++/QQ8deyVh3ZKI7yzQ8ddJc+Uz+9/lX7Nm91/N2L2hYm6SE+GPWXdy4LrGBvK/Ruck12bbvsOe5AGr9pjmHNm3lcHpx+54ix5XtIt+J9JBLIiJVRKRa/m2gE7DquKe9D9wYmm3RBthb1vFjKKUgi8iXhW7/CXgJqAY8JCLDytpoOJJT6rE5PbPgfm39GDoAABJsSURBVHrGFpKT60WyyWJd++DNvPfEJHIjfIVu17n0mbho+srNXHJGXV/arntNO7a9t8iXtl3bLnJEw15KcRqwUERWAF8Cs1R1jojcJiK3hZ4zG9gArAdeAwafTPbSZlnEFbo9ALhcVX8UkWeBxcCTRb0oNPg9AEBiqhMIVDmZjL4659Lz2bdzLz+s2siZbc72O45x1GtfrCUmIHQ5u8z7c8pM4mKo0+kC0h570/O2XVRe3SZV3QCcV8T6VwvdVqDcBs1LK8gBEalJXk9aVPXHUIgDIpJT3ItCA+NjAGLjU8r088nM2ErDBskF9xuk1Cczc2tZ3uqk/KpVU879XSvO6diS2ErxJFRN4Obn72DCXf/wPIvfXPlMXDNj5WY+S9vOP/u0QcT7EynUvqwl+1duJPtH74dxwL3toiIfqVdaQa4OfAUIoPnz60SkamhdxCxZupwmTRrTqFFDMjK20rv31dxwo/d7b2c8PYUZT08B4Mw2Z3P5n7qdksUY3PlMXLJow3Ymfrmesf0uJiHOn2n9p/k4XAHubRe5Ffi60yVuQaraqJiHcoFryj1NIcFgkCF3jmD2rDeJCQSYMHEqqalrI9mkcyr1vINAo2ZIYjUS7n6J7I/fRg/tJ77LzUhiEpWvG0pw6yaOTCpy5KjcufKZvDz2Gdq2u5BatWuwdNV8nn3yZd6a9G7E2x32/lcs3byTPYeO0mn0Rwy6pCnjF6/jaDCX26YtBuDc+jUZ0fnciGfJF0isRK325/LdPcfP1vKOK9tFvopbjvOGISLaQFmHLMrTn5Lb+R0BgGdvjvE7AtUfX+B3BMCNK4asf/ZKvyMA8IUDVwzptNu/HnZhOUczTvov73sa9Qu75jy7aYpTJ+u0Q6eNMVElWIH7yFaQjTFRJZp36hljTIWi1kM2xhg3WA/ZGGMcEbXT3owxpqKpuOXYCrIxJsrkVOCSbAXZGBNVbKee45Yc8ed0lceL/f0IvyPw99fdOAX2ny/d5ncEZ7hyUEa0sJ16xhjjCOshG2OMI6yHbIwxjghW4AtJWEE2xkQVm4dsjDGOsDFkY4xxhI0hG2OMI2zIwhhjHGFDFsYY44iKPMvCjcO2itG5UwdWr1rAd6kLGXqvPxdNPC25Lq++/SLTPv0XUz95g7639vKs7QdHT+a3fxzONXc/8bPHJn7wX8699i/sztrvWR6A2xe+wJ/mPsmtsx/nlg8e8azdhFvuodqL/6bqI68VrJMq1Ui85ymqPjmBxHuegsSqEc3w0IfL6fjSXHqO/6Rg3ciPU+kx9r9c+/on3PXeErIOZ0c0w/Fc+I64lAPyhizCXVzjbEEOBAKMevExuna7nubndaRPnx40a3am5zlycoI8//eX6f3bG+h/1UCuvfn3ND6rkSdtd+9wEa/cP+hn67fu2M0XK76jfp2anuQ43qS+jzK2y3DGd3vAszaPLpzLgZF/PWZdpS59CaYuY/+wmwmmLqPyVX0jmqH7OQ0Z3euiY9a1aVSHt2/pwL/7d+D0mlUYv3hdRDMU5sp3xJUc+XJPYHFNiQVZRC4SkaTQ7QQR+buIfCAiT4lI9UgGa31hS9LSNrFx4w9kZ2czbdoMunfrHMkmi7Rz+07WrMy7gu7BA4fYtG4TdevV8aTtVmc3oXrVxJ+tf3rCu9x1/dWIOHV9xogKrl2J7t93zLrYlhdzdNE8AI4umkdsy8hezPaChrVJSog/Zt3FjesSG8j7Gp2bXJNt+w5HNENhrnxHXMmRT0/gP9eU1kMeDxwM3X4RqA48FVr3egRzkZxSj83pmQX30zO2kJxcL5JNlqp+g3o0bX4Wq75O9S3Dx0u+oW6tGjRtlOJTAuW6ScO4ZeajtOzX0acMeQLVa6J7d+Wl2ruLQHV//mLIN33lZi45o65n7bnyHXElR76KPGRR2k69gKrmhG63UtXzQ7cXisjy4l4kIgOAAQASU51AoMrJJ/VZQmICT497lOceHMWB/QdLf0EEHDpylNfe/Yh/jhjsS/sAb/R8mH3bdpNYO4nrJg1jR9oWNn/5nW95juHjzpzXvlhLTEDocrZfvyhNPo3inXqrRKR/6PYKEWkFICJnAcXuvVDVMaraSlVblbUYZ2ZspWGD5IL7DVLqk5m5tUzvdbJiYmN4etyjzHn3Iz6evcCXDACbt+4gY/tOrr33Ka4Y/De27dxDn6HPsGN3lmcZ9m3bDcDBnVmsmbuU5BZneNb28XL37kaq1wJAqtciN2uPLzlmrNzMZ2nbebxrS0+HkVz5jriSI18QDXspiYg0FJGPRSRVRFaLyJAintNBRPaKyPLQ8uDJZC+tIN8K/FZE0oCzgS9EZAPwWuixiFmydDlNmjSmUaOGxMXF0bv31Xwwc14kmyzWgyOHsXHdJib/c6ov7ec76/RkPh33OHNG/405o//GabVrMPXpe6lTM8mT9uMSKhFfpXLB7TPaN+fHNemetF2UnOVfEN+uEwDx7TqRs+xzzzMs2rCdiV+u54XfX0hCnLezSF35jriSI185DlnkAP9PVc8G2gC3i8jZRTzvM1VtEVoePpnsJW5BqroXuDm0Y69x6Pnpqhrxs4sHg0GG3DmC2bPeJCYQYMLEqaSmro10sz9zXuvmXHXtFaxLTWPyR+MBGP3EGBb9d3HE2x76wgSWrl7Pnn37+d3ABxjcuwu/v6xtxNstTpU6SfQacxcAgdgYVs/4nA2ffuNJ2wkDhxP76/OQqtWp9twUDk+fyJFZb5E4eARx7a9Ad2zn4CuRnYY37P2vWLp5J3sOHaXT6I8YdElTxi9ex9FgLrdNy9sezq1fkxGdz41ojnyufEdcyZGvvIYsVHULsCV0e5+IfAukABHbiSSRHm+JjU/xfUCnRW3//qwubNFc/68Y8my3N/yOALhxxZD4S1r4HQGAaoOm+B3BGTlHM056zKdjg8vDrjkfp38UVnsi0ghYAJyjqlmF1ncA3gHSgUzgHlVdfQJxj2FH6hljosqJTGcrPAEhZIyqjjnuOVXJK7p3Fi7GIV8Dp6vqfhHpAkwHyjwJ2wqyMSaqnMih06HiO6a4x0UkjrxiPFlV3y3i9VmFbs8WkdEiUkdVd5xY6jxWkI0xUaW85hdL3pSZccC3qjqymOfUA7apqopIa/ImSuwsa5tWkI0xUaUcD/hoB9wArCx03MVw4JcAqvoq0AsYJCI5wCGgr57EjjkryMaYqFKOsywWAiXu9FPVl4CXyqVBrCAbY6KMi4dEh8sKsjEmqrh40qBwWUE2xkSVoLp4Ys3wnBIFefnODX5HAGDqlf/yOwJ3j/fvSL/Cjr413e8IdkBGlKrIJxc6JQqyMebUYWPIxhjjCBtDNsYYR+TakIUxxrjBesjGGOMIm2VhjDGOsCELY4xxhA1ZGGOMI6yHbIwxjqjIPeTSLnLqq86dOrB61QK+S13I0HtvP6VzxCUl8tsxf+HqT5+m+ydPUeeCJp60+9Dk/9Bx+Fh6PjG5YN3LsxZz7ZNv0vupKdz28nS2790f8RwJt9xDtRf/TdVHXitYJ1WqkXjPU1R9cgKJ9zwFiVUjnqMwF7YLFzK4lAMgqMGwF9c4W5ADgQCjXnyMrt2up/l5HenTpwfNmpX5yigVPkfrh28g4+NvmPHbocy8fDh712V60m73i5oxelD3Y9bddOn5/HvYdUy7rx/tz2nMmDlLIp7j6MK5HBj512PWVerSl2DqMvYPu5lg6jIqX9U34jnyubBduJDBpRz5VDXsxTUlFmQR+YuINPQqTGGtL2xJWtomNm78gezsbKZNm0H3bp1PyRxx1RKoe1FT1k/5BIDc7CDZWQc9afuCJikkJVY+Zl3VhPiC24eOZJd8wthyEly7Et2/75h1sS0v5uiivMvNH100j9iW7TxIkseF7cKFDC7lyJeLhr24prQe8iPA/0TkMxEZLCK/8CIUQHJKPTan/9QLTM/YQnJyPa+adypH1V/+giM793Hx8wPoOvdR2j5zK7EJlTzNcLx/zPyCzg++zuyv1jCoSxtfMgSq10T37gJA9+4iUL2mZ227sF24kMGlHPmitocMbAAakFeYLwBSRWSOiNwkItWKe5GIDBCRpSKyNDf3QDnGPTUFYmKo1bwRa9+Yz8zOI8g5eIRz/tzN10x3dG3L3If70+WCprz12QpfsxRw8AtmvJerGvbimtIKsqpqrqrOU9U/AsnAaOAK8op1cS8ao6qtVLVVIFClTMEyM7bSsEFywf0GKfXJzNxapvc6GS7kOLBlFwe37GLHsjQAvp/1JbWaN/I0Q3G6tGrK/BVpvrSdu3c3Ur0WAFK9FrlZezxr24XtwoUMLuXIpyfwn2tKK8jHDA+qaraqvq+q/YDTIxcLlixdTpMmjWnUqCFxcXH07n01H8ycF8kmnc1x+Me9HMjcRdKv6gNQ/5L/Y+/aDE8zFPb99p8K3ycrN9C4rndDBYXlLP+C+HadAIhv14mcZZ971rYL24ULGVzKkS+ouWEvriltHnKf4h5Q1YjuVQoGgwy5cwSzZ71JTCDAhIlTSU1dG8kmnc7x5QMTueQfg4iJi2XfD9v5/O4xnrQ7bMIclq7PYM/+w3R6YDyDulzEwtTv2bR9NwER6tesxv19OkY8R8LA4cT++jykanWqPTeFw9MncmTWWyQOHkFc+yvQHds5+MojEc+Rz4XtwoUMLuXI5+LYcLgk0uFj41Mq7k+nnI3/ReQLV2muHe/PDrjjuXDFkNqTv/U7gjlOztGMk560U6vamWHXnF371nkxSShsdqSeMSaqVOQeshVkY0xUcXF+cbisIBtjoor1kI0xxhEuzp4IlxVkY0xUcfGAj3BZQTbGRJWKPGTh7NnejDGmLMrzSD0RuUJE1ojIehEZVsTjlURkaujx/4lIo5PJbgXZGBNVyuvkQiISA7wMXAmcDfQTkbOPe9ofgd2q2gR4HnjqZLJbQTbGRJVyPLlQa2C9qm5Q1aPAW8DVxz3namBi6PbbwGUiUvaDTU7kt4lfCzDAMriTw4UMruRwIYMrOVzIUJbMwNJCy4BCj/UCxha6fwPw0nGvXwU0KHQ/DahT1jwVpYc8wO8AuJEB3MjhQgZwI4cLGcCNHC5kOCFa6MyUocWbk8QUo6IUZGOM8VoGUPiKSQ1C64p8jojEAtWBnWVt0AqyMcYUbQlwpog0FpF4oC/w/nHPeR+4KXS7F/BfDY1dlEVFmYfs658RIS5kADdyuJAB3MjhQgZwI4cLGcqNquaIyJ+BuUAMMF5VV4vIw8BSVX0fGAf8S0TWA7vIK9plFvHTbxpjjAmPDVkYY4wjrCAbY4wjnC7IpR226FGG8SKyXURW+dF+KENDEflYRFJFZLWIDPEpR2UR+VJEVoRy/N2PHKEsMSKyTERm+phhk4isFJHlIrLUpww1RORtEflORL4VkbY+ZGga+hnkL1kicqfXOaKBs2PIocMW1wKXA+nk7fHsp6qpHudoD+wH3lDVc7xsu1CG+kB9Vf1aRKoBXwE9fPhZCFBFVfeLSBywEBiiqou9zBHKcjfQCkhS1a5etx/KsAlopao7/Gg/lGEi8Jmqjg3NBEhUVe8uv/3zPDHkTQW7SFW/9ytHReVyDzmcwxYjTlUXkLf31DequkVVvw7d3gd8C6T4kENVdX/oblxo8fw3uog0AK4CxnrdtktEpDrQnrw9/ajqUT+LcchlQJoV47JxuSCnAJsL3U/HhyLkmtDZpFoC//Op/RgRWQ5sBz5SVT9yvAAMBfw+E7kC80TkKxHx4yi1xsCPwOuh4ZuxIlLFhxyF9QWm+JyhwnK5IJvjiEhV4B3gTlXN8iODqgZVtQV5Ry21FhFPh3FEpCuwXVW/8rLdYlyiqueTdzaw20PDW16KBc4HXlHVlsABwJd9LQChIZPuwL/9ylDRuVyQwzls8ZQRGrN9B5isqu/6nSf0p/HHwBUeN90O6B4av30LuFREJnmcAQBVzQj9ux14j7xhNi+lA+mF/kp5m7wC7Zcrga9VdZuPGSo0lwtyOIctnhJCO9PGAd+q6kgfc/xCRGqEbieQt8P1Oy8zqOpfVbWBqjYib5v4r6pe72UGABGpEtrBSmiYoBN5Z/7yjKpuBTaLSNPQqssAT3f0HqcfNlxxUpw9dLq4wxa9ziEiU4AOQB0RSQceUtVxHsdoR96p/1aGxm8BhqvqbI9z1AcmhvakB4BpqurbtDOfnQa8Fzr1bSzwpqrO8SHHHcDkUKdlA9Dfhwz5v5QuBwb60X60cHbamzHGnGpcHrIwxphTihVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxhBVkY4xxxP8HBhsgf+LWQBYAAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"digit = X[0].reshape([8, 8])\n",
"sns.heatmap(digit, annot=True);"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"0"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y[0]"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们需要划分一个训练集与验证集，在训练集上训练 `Step 1`得到的机器学习模型，在验证集上预测，预测值会被处理为返回的损失值 `loss` 。首先我们要对原数据进行切分以得到训练集和验证集："
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"X_train, X_test, y_train, y_test = train_test_split(\n",
" X, y, test_size=0.33, random_state=42)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"然后我们需要在训练集`X_train`，`y_train`上对机器学习模型`ML_model`进行训练："
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
" intercept_scaling=1, loss='squared_hinge', max_iter=600,\n",
" multi_class='ovr', penalty='l2', random_state=42, tol=0.0001,\n",
" verbose=0)"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ML_model.fit(X_train, y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"#### Step 3. 在验证集上得到相应的评价指标"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"为了保证我们的机器学习模型具有对未知数据的泛化能力，所以需要用未参与训练的验证集来评价之前得到的机器学习模型。\n",
"\n",
"所以，我们需要一个评价指标。这里我们选用最简单的 `accuracy_score`"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"y_pred = ML_model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"0.9444444444444444"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"accuracy = accuracy_score(y_test, y_pred)\n",
"accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们注意到本示例的 `Step 2`与`Step 3`的训练与验证部分可能存在偏差。比如如果数据集划分得不均匀，可能会在验证集上过拟合。\n",
"\n",
"我们可以用**交叉验证**(Cross Validation, CV)的方法解决这个问题，交叉验证的原理如图所示："
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"![](\"https://scikit-learn.org/stable/_images/grid_search_cross_validation.png\")
"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from sklearn.model_selection import StratifiedKFold # 采用分层抽样\n",
"from sklearn.model_selection import cross_val_score"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"因为这只是一个简单的示例，所以采用3-Folds交叉验证以减少计算时间。\n",
"\n",
"在实践中建议使用5-Folds交叉验证或10-Folds交叉验证。"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"array([0.92654424, 0.94490818, 0.96327212])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"scores = cross_val_score(ML_model, X, y, cv=cv)\n",
"scores"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们采用交叉验证的平均值作为最后的得分："
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"0.9449081803005009"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"score = scores.mean()\n",
"score"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"#### Step 4. 对评价指标进行处理，使其越小越好，返回loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们注意到，评价指标正确率的取值范围是0-1，且是越大越好的，所以我们需要将score处理为越小越好的loss："
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"0.055091819699499056"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"loss = 1 - score\n",
"loss"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"#### 用一个函数实现AutoML评价器"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"metric = \"accuracy\"\n",
"def evaluate(config: dict) -> float:\n",
" layered_dict = layering_config(config)\n",
" display(plot_layered_dict(layered_dict)) # 用于在jupyter notebook中可视化，实践中可以删除次行\n",
" AS_HP = layered_dict['classifier'].copy()\n",
" AS, HP = AS_HP.popitem()\n",
" ML_model = eval(AS)(**HP)\n",
" # 注意到： X, y, cv, metric 都是函数外的变量\n",
" scores = cross_val_score(ML_model, X, y, cv=cv, scoring=metric)\n",
" score = scores.mean()\n",
" print(f\"accuracy: {score:}\") # 用于在jupyter notebook中调试，实践中可以删除次行\n",
" return 1 - score"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"accuracy: 0.9449081803005009\n"
]
},
{
"data": {
"text/plain": [
"0.055091819699499056"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"evaluate(CS.get_default_configuration())"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"#### 用一个类实现AutoML评价器"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们注意到，`evaluate`函数的 X, y, cv, metric 都是函数外的变量，不利于管理，所以实践中我们一般采用定义**类**来实现评价器。"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"首先我们定义一个类，这个类需要根据 训练数据、评价指标和交叉验证方法 等条件进行初始化："
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"default_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)\n",
"\n",
"class Evaluator():\n",
" def __init__(self, \n",
" X, y, \n",
" metric=\"accuracy\", \n",
" cv=default_cv):\n",
" # 初始化\n",
" self.X = X\n",
" self.y = y\n",
" self.metric = metric\n",
" self.cv = cv"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"然后我们需要实现`__call__`魔法方法，在该方法中实现整个评价过程："
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"default_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)\n",
"\n",
"class Evaluator():\n",
" def __init__(self, \n",
" X, y, \n",
" metric=\"accuracy\", \n",
" cv=default_cv):\n",
" # 初始化\n",
" self.X = X\n",
" self.y = y\n",
" self.metric = metric\n",
" self.cv = cv\n",
" \n",
" def __call__(self, config: dict) -> float:\n",
" layered_dict = layering_config(config)\n",
" AS_HP = layered_dict['classifier'].copy()\n",
" AS, HP = AS_HP.popitem()\n",
" ML_model = eval(AS)(**HP)\n",
" scores = cross_val_score(ML_model, self.X, self.y, cv=self.cv, scoring=self.metric)\n",
" score = scores.mean()\n",
" return 1 - score"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"实例化一个评价器对象："
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"evaluator = Evaluator(X, y)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"用配置空间的一个采样样本测试这个AutoML评价器："
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"text/plain": [
"0.017250973845297835"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"evaluator(CS.sample_configuration())"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"### 根据上述知识实现一个简单的AutoML系统"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"**配置空间** 、**评价器**我们都有了，UltraOpt已经实现了成熟的**优化器**，所以我们只需要用`ultraopt.fmin`函数将这些组件串起来"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"from ultraopt import fmin"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"100%|██████████| 40/40 [00:14<00:00, 2.67trial/s, best loss: 0.012]\n"
]
},
{
"data": {
"text/plain": [
"+----------------------------------------------------------------------------+\n",
"| HyperParameters | Optimal Value |\n",
"+-----------------------------------------------------+----------------------+\n",
"| classifier:__choice__ | KNeighborsClassifier |\n",
"| classifier:KNeighborsClassifier:n_neighbors | 4 |\n",
"| classifier:KNeighborsClassifier:p | 2:int |\n",
"| classifier:KNeighborsClassifier:weights | distance |\n",
"| classifier:LinearSVC:C | - |\n",
"| classifier:LinearSVC:dual | - |\n",
"| classifier:LinearSVC:loss | - |\n",
"| classifier:LinearSVC:max_iter | - |\n",
"| classifier:LinearSVC:multi_class | - |\n",
"| classifier:LinearSVC:penalty | - |\n",
"| classifier:LinearSVC:random_state | - |\n",
"| classifier:RandomForestClassifier:bootstrap | - |\n",
"| classifier:RandomForestClassifier:criterion | - |\n",
"| classifier:RandomForestClassifier:max_features | - |\n",
"| classifier:RandomForestClassifier:min_samples_leaf | - |\n",
"| classifier:RandomForestClassifier:min_samples_split | - |\n",
"| classifier:RandomForestClassifier:n_estimators | - |\n",
"| classifier:RandomForestClassifier:random_state | - |\n",
"+-----------------------------------------------------+----------------------+\n",
"| Optimal Loss | 0.0117 |\n",
"+-----------------------------------------------------+----------------------+\n",
"| Num Configs | 40 |\n",
"+-----------------------------------------------------+----------------------+"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"result = fmin(evaluator, HDL, optimizer=\"ETPE\", n_iterations=40)\n",
"result"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们可以对AutoML得到的结果进行数据分析：\n",
"\n",
"首先， 我们可以绘制拟合曲线："
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"pycharm": {}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEYCAYAAACdnstHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xcZZ3n8c+3u7o6fUknIU2aSxISLgtEBGLCzYkKAhrckeAYR1ARRxxeqzI6yzorOsqwLK6DrqKueEFwREQDg6JRUOSSRlFuCQkkIQRDQkICJAY6JH1Jqi+//eOcSk4q1d11qrtuXb/363VedS7PeepXp7vr6ec5z3kemRnOOedcPmpKHYBzzrnK5YWIc865vHkh4pxzLm9eiDjnnMubFyLOOefy5oWIc865vHkh4pwbkqQZkkxSotSxuPLjhYiraJI+IGmppE5JL0v6raR5pY6rWkm6WtJPSh2HKx4vRFzFknQF8A3g/wBtwHTgO8CCUsYV5f+9u7HOCxFXkSRNAK4BPmlmvzCzLjPrNbNfm9m/hGnqJX1D0kvh8g1J9eGxMyVtlvQ/JG0LazH/EB47TdIrkmoj7/ceSU+H6zWSrpT0vKRXJd0h6aDwWLrp51JJm4AHw/0flrQxTP9FSS9IOidGfpdI2iRpu6R/jcRVK+nz4bm7JC2TNC08dpyk+yS9JmmtpL8f4nq2S/qypMcl7ZT0q3QMWdIeJmlxmO86Sf8Y7p8PfB54f1gzfCqvH66rKF6IuEp1BjAOuGuINP8KnA6cDJwEnAp8IXL8EGACcDhwKXCDpElm9hjQBbw9kvYDwE/D9X8CLgDeBhwGdAA3ZLz324DjgXdKmkVQQ/ogcGjkPdNyyW8ecCxwNnCVpOPD/VcAFwHvAlqAjwLdkpqA+8KYpwAXAt8JYxnMh8PzDwX6gG8Nkm4RsDmMdSHwfyS93cx+R1ArvN3Mms3spCHey40VZuaLLxW3EHwhvzJMmueBd0W23wm8EK6fCfQAicjxbcDp4fq1wA/D9fEEhcoR4fYa4OzIeYcCvUACmAEYcGTk+FXAzyLbjUAKOCdGflMjxx8HLgzX1wILsnz29wN/zNj3feDfBrlW7cC/R7ZnhTHWRmJIANOAfmB8JO2XgR+F61cDPyn174cvxVu8vdZVqleBVkkJM+sbJM1hwMbI9sZw3948Ms7tBprD9Z8Cf5b0ceDvgCfNLJ3XEcBdkgYi5/YT3JdJezEjjr3bZtYt6dXI8Vzye2WQOKcRFJaZjgBOk7Qjsi8B3JolbbaYNwJ1QGtGmsOA18xsV0bauUPk68Ywb85yleoRYA9BM9BgXiL4Mk2bHu4blpk9Q/DleB77N2VB8GV7nplNjCzjzGxLNIvI+svA1PSGpAZgcsz8BvMicNQg+x/KyLPZzD4+RF7TIuvTCWpD2zPSvAQcJGl8Rtp0rD4seJXxQsRVJDN7naCZ6AZJF0hqlFQn6TxJXwmT/Qz4gqSDJbWG6eN0P/0p8GngrcB/RvZ/D/iSpCMAwvyH6hF2J/BuSW+WlCRo8tEI8ou6Cfjfko5R4ERJk4HfAP9F0sXhdamTdErkXko2H5I0S1IjQaeFO82sP5rAzF4E/gx8WdI4SScS3E9KX9etwAxJ/t1SJfwH7SqWmX2N4MbyF4C/Evz3fTnwyzDJtcBS4GlgJfBkuC9XPyO42f2gmUX/I/8msBj4vaRdwKPAaUPEuZrg5vkiglpJJ8H9lz355Jfh68AdwO+BncDNQEPY3PQOghvqLxE0h10H1A+R163Aj8K044BPDZLuIoL7JC8RdGz4NzO7PzyWLmxflfRkjp/BVTCZee3TuWKS1AzsAI4xsw2ljgeCLr4EN8RvKnUsrrJ4TcS5IpD07rDJrQn4vwQ1oxdKG5VzI+eFiHPFsYCg+ecl4BiCLrreDOAqnjdnOeecy5vXRJxzzuWt6h42bG1ttRkzZgybrquri6ampsIHFFO5xgXlG5vHFV+5xuZxxTOacS1btmy7mR18wIFSPzJf7GXOnDmWiyVLluSUrtjKNS6z8o3N44qvXGPzuOIZzbiApZblO9Wbs5xzzuXNCxHnnHN580LEOedc3rwQcc45lzcvRJxzzuXNCxHnnHN580LEOedc3rwQcc45l7eqe2J9pHZ0p9iwvYudPb20NNQxs7WJiY3JUTl3JHmXUjruzj19LN/UsV/cI/1MQ51fyrxHcnyo61VqpYqtkH9XrrC8JhLDju4Uyzd1kOobYFJjklTfAMs3dbCjOzXic0eSdylF466t0X5xj/QzDXV+KfMeyfGhrleplSq2Qv5ducLzQiSGDdu7aEwmuGPpZpas/SuNyQSNyQQbtnflfG5jMoGkA84d7ni5isYN7Bf3SD/TUOeXMu+RHB/qepVaqWIbyc+yUv9uxhIvRGLY2dNLQ10tf/zLX3lq8w4AGupq2dnTm/O5UdFzhzue/o8r3cxQLv9pDRX3cJ+p0HkPdc1GkvdIjo/0mhRSqWIbyfuW8/WsFkUrRCTNl7RW0jpJV2Y5Xi/p9vD4Y5JmZByfLqlT0mci+16QtFLSCklLC/0ZWhrq6OntZ1xdLT2pfgB6evtpaajL+dyo6LlDHS/nJpCh4h7uMxcy7+Gu2UjyHsnxkV6TQipVbCN533K+ntWiKIWIpFrgBuA8YBZwkaRZGckuBTrM7GjgeuC6jONfB36bJfuzzOxkM5s7ymEfYGZrE92pPuoTNfT09tOd6qM71cfM1uGHWk6f253qw8wOOHeo4+XcBBKNG9gv7uE+c5y8M88fLu/hrtlI8h7J8aGuV6mVKraR/J6M9HfMjVyxaiKnAuvMbL2ZpYBFBNOFRi0AbgnX7wTOliQASRcAG4DVRYo3q4mNSWZPn0RjspbO3b0kEzXMnj4pp54g6XOTiRo6ulMHnDvU8XKuskfj7h+w/eIe7jPHyTvz/OHyHu6ajSTvkRwf6nqVWqliG8nvyUh/x9zIFWV6XEkLgflm9rFw+2LgNDO7PJJmVZhmc7j9PHAasBu4DzgX+AzQaWb/N0yzAegADPi+md04yPtfBlwG0NbWNmfRokXDxtzZ2Ulzc3PWY/9v+W62dg1w7bzGHD79yPWk+hkAagS9u7upG9fIgAX/ATQka4c7vWiGumbFVgnXrJyuV6Zyjc3jimc04zrrrLOWZWvxqYTnRK4GrjezzrBiEjXPzLZImgLcJ+lZM/tDZqKwcLkRYO7cuXbmmWcO+6bt7e0Mlm7xthW8sv61QY+PtnT7fmMywWvPr+Cgo06mO9VXdv9xDXXNiq0Srlk5Xa9M5RqbxxVPMeIqVnPWFmBaZHtquC9rGkkJYALwKkFt5CuSXgD+Gfi8pMsBzGxL+LoNuIug2azgmusTdIXtxsVQzk0g5cqvmXPFUayayBPAMZJmEhQWFwIfyEizGLgEeARYCDwYTsn4lnQCSVcTNGd9W1ITUGNmu8L1dwDXFPyTAE31Cbr39A+fcBQFX4pJXl+fYPb0SaOa91h94reQ18w5FyhKTcTM+oDLgXuBNcAdZrZa0jWSzg+T3QxMlrQOuAI4oBtwhjbgYUlPAY8Dd5vZ7wrzCfbXlKwl1T9Aqm+gGG9XUP7Er3NuJIp2T8TM7gHuydh3VWR9N/C+YfK4OrK+HjhpdKPMTVN9cNm69vSRTFT2f+zZusKm98+eXtmfzTlXeJVwY73s7C1EUn1MaqrsL9qdPb1Mymi6aqirpaNINZGx2pTmXLXwYU/y0JRM10SKe1+kEEr5xK83pTlX+bwQyUNTffCcQeee4vXQKpRSPvHrg+c5V/m8EMlDc+SeSKUr5RO/5fwkvnMuN35PJA/peyLdRXxWpJDSXWGLLd2Ulr6ZDz54nnOVxmsieUjfE+kcA/dESskHz3Ou8nkhkof0PZGx0JxVSj54nnOVz5uz8hDt4utGplRNac650eE1kTzUJ2qorZHXRJxzVc8LkTxIoilZOyaeE3HOuZHwQiRPzfWJMfGciHPOjYQXInlqqk+MmS6+zjmXLy9E8tRYn/Auvs65queFSJ6a62v9xrpzrup5IZKnpmTCCxHnXNXz50Ty1FTkKXKdc8XnUxUMz2sieWqq9y6+zo1lPlVBbrwQyVOTd/F1bkzzqQpy44VInpqTCVJ9A/T2V/486865A/lUBbkpWiEiab6ktZLWSboyy/F6SbeHxx+TNCPj+HRJnZI+k2uehdSYHg7em7ScG5NKOetnJSlKISKpFrgBOA+YBVwkaVZGskuBDjM7GrgeuC7j+NeB38bMs2Ca07Mb+s1158Ykn6ogN8WqiZwKrDOz9WaWAhYBCzLSLABuCdfvBM6WJABJFwAbgNUx8yyYvRNT+X0R58Ykn6ogN8Xq4ns48GJkezNw2mBpzKxP0uvAZEm7gc8C5wKfyZZ+iDwBkHQZcBlAW1sb7e3twwbc2dk5ZLrn/xoUHn945HG2TKwdNN1oGy6uUirX2Dyu+Mo1tlLG9TqwYn32Y9V8vSrhOZGrgevNrDOsmMRmZjcCNwLMnTvXzjzzzGHPaW9vZ6h0jRte4/plj3DsG05i3jGtecWVj+HiKqVyjc3jiq9cY/O44ilGXMUqRLYA0yLbU8N92dJslpQAJgCvEtQuFkr6CjARGAhrJ8tyyLNg0rMbejdf51w1K1Yh8gRwjKSZBF/0FwIfyEizGLgEeARYCDxoZga8JZ1A0tVAp5l9OyxohsuzYJrT90T8xrpzrorlfGNd0vskjQ/XvyDpF5LelMu5ZtYHXA7cC6wB7jCz1ZKukXR+mOxmgnsg64ArgCG77A6WZ66fZ6Qak+EUuV4Tcc5VsTg1kS+a2X9KmgecA3wV+C6D3MzOZGb3APdk7Lsqsr4beN8weVw9XJ7Fkq6J+HDwzrlqFqeLb/rb8r8CN5rZ3UDV9nUbV1dDjbw5yzlX3eIUIlskfR94P3CPpPqY548pwTzrPn6Wc666xSkE/p7g/sM7zWwHcBDwLwWJqkI01fucIs656pbzPREz6wZ+Edl+GXi5EEFVCh8O3jlX7XIuRMLmq/cCM6Lnmdk1ox9WZWj2iamcc1UuTu+sXxE8+b8M2FOYcCpLo0+R65yrcnEKkalmNr9gkVSgpvoEW3b0lDoM55wrmTg31v8s6Y0Fi6QCNdfXehdf51xVi1MTmQd8RNIGguYsAWZmJxYksgrQ6L2znHNVLk4hcl7BoqhQzT7PunOuyuXcnGVmGwlG0X13uEwM91WtpmSC3b0D9Pk86865KhVnAMZPA7cBU8LlJ5L+qVCBVYL0cPDdvf6siHOuOsVpzroUOM3MugAkXUcwbPv/K0RglSA9RW7Xnj5axtWVOBrnnCu+OL2zxL5BGAnX85tqcIyIFiLOOVeN4tRE/gN4TNJd4fYFBHOAVK3msDnLhz5xzlWrOGNnfV3SQ8DfhLv+wcyWFyasyuATUznnql2s6XHNbBnBsCeO6MRUXog456rTsIWIpIfNbJ6kXYBFDxE8bNhSsOjK3N57Iv7UunOuSg1biJjZvPB1fOHDqSxNfk/EOVfl4jwncl0u+4Y4f76ktZLWSboyy/F6SbeHxx+TNCPcf6qkFeHylKT3RM55QdLK8NjSXGMZLU1+T8Q5V+XidPE9N8u+nIZCkVQL3BCmnwVcJGlWRrJLgQ4zOxq4HkgXUKuAuWZ2MjAf+L6kaA3qLDM72czm5v5RRkdjshbJCxHnXPUathCR9HFJK4FjJT0dWTYAK3N8n1OBdWa23sxSwCJgQUaaBcAt4fqdwNmSZGbdZpb+lh7H/vdlSio9z3pXypuznHPVSWZDfydLmgBMAr4MRJuhdpnZazm9ibQQmG9mHwu3LyZ4+v3ySJpVYZrN4fbzYZrtkk4DfggcAVxsZneFaTYAHQQFy/fN7MZB3v8y4DKAtra2OYsWLRo25s7OTpqbm4dN989Lujnx4Fo+ekL9sGlHQ65xlUK5xuZxxVeusXlc8YxmXGedddayrC0+ZpbzQlCYnAq8Nb3keN5C4KbI9sXAtzPSrCKY+Cq9/TzQmpHmeOBxYFy4fXj4OgV4Kpd45syZY7lYsmRJTunO+uoS++Rty3JKOxpyjasUyjU2jyu+co3N44pnNOMCllqW79Q4N9Y/BvwBuBf4X+Hr1TmevgWYFtmeGu7Lmia85zEBeDWawMzWAJ3ACeH2lvB1G3AXQQFXVE0+p4hzrorFubH+aeAUYKOZnQXMBnbkeO4TwDGSZkpKAhcCizPSLAYuCdcXAg+amYXnJAAkHQEcB7wgqUnS+HB/E/AOgtpMUTXV1475eyI7ulMs39TBQ2u3sXxTBzu6U6UOyTlXJuIUIrvNbDcE3XHN7Fng2FxOtODG+OUEtZc1wB1mtlrSNZLOD5PdDEyWtA64gn33X+YBT0laQVDb+ISZbQfagIclPUXQxHW3mf0uxucZFU3JsV0TSRcgqb4BJjUmSfUNeEHinNsrzrAnmyVNBH4J3CepA8h5Uiozuwe4J2PfVZH13cD7spx3K3Brlv3rgZNyjr5Axnpz1obtXTQmE3vHCUu/btjexezpyVKG5pwrAzkVIpIEfMrMdgBXS1pCcM+i6P/5l5um+rHdxXdnTy+TGvcvLBrqaunwmohzjhwLkfDexD3AG8PthwoaVQVpStaO6ZpIS0MdPb39e2sgAD29/bQ0+CRczrl490SelHRKwSKpUE31CbpT/QwMlM0zkKNqZmsT3ak+ulN9mNne9ZmtTaUOzTlXBuLcEzkN+JCkF4Au9o3ie2IhAqsUzZGRfMePwSlyJzYmmT19Ehu2d9HRnaKloY5jD5nExEa/H1IqO7pTbNjexc6eXloa6pjZ2uQ/jxHw6zkycQqRdxYsigqWHg6+O9U/JgsRSBck/kdVDtK95RqTCSY1Junp7Wf5pg5mT/eCPR9+PUcuTnPWJuAtwCVmtpFgqJG2gkRVQdLDwfvEVK4Yor3lJO1d37C9q9ShVSS/niMXpxD5DnAGcFG4vYtgZN6q5sPBu2La2dNLQ13tfvsa6mrZ2dNboogqm1/PkYtTiJxmZp8EdgOYWQdQ9fW9vbMb+sRUrgjSveWivLdc/vx6jlycQqQ3nBfEACQdDAwUJKoKsm92Q6+JuMLz3nKjy6/nyMUpRL5FMOzIFElfAh4mGB6+qvk8666Y0r3lkokaOrpTJBM1fhN4BPx6jlzOvbPM7DZJy4CzCbr3XhCOqlvV0l18/ca6KxbvLTe6/HqOTM6FiKTrzOyzwLNZ9lWtvV18/Z6Ic64KFWWO9bGssc67+DrnqtewNRFJHwc+ARwp6enIofHAnwoVWKWoqRGNY3z8LOecG0wuzVk/BX7LCOZYH+vG+ki+zjk3mGELETN7HXidfQ8ZugxjfSRf55wbTC7NWQ+b2TxJuwifEUkfIhiAsaVg0VWIsT4xlXPODSaXmsi88HV84cOpTE31Cb+x7pyrSnF6Z7lBNIdzijjnXLUpWiEiab6ktZLWSboyy/F6SbeHxx+TNCPcf6qkFeHylKT35JpnsXjvLOdctSpKIRKOuXUDwXMls4CLJM3KSHYp0GFmRwPXA9eF+1cBc83sZGA+8H1JiRzzLIpmb85yzlWpnAsRSW+XdLOkr0n6B0lzJNXnePqpwDozW29mKWARsCAjzQLglnD9TuBsSTKzbjNLf0OPY9/N/VzyLIomb85yzlWpODWRHwK/Bh4FjgSuAlbneO7hwIuR7c3hvqxpwkLjdWAygKTTJK0GVgL/LTyeS55F0ZSspSscBdQ556pJnOlxN5rZL8P1/yxEMIMxs8eAN0g6HrhF0m/jnC/pMuAygLa2Ntrb24c9p7OzM6d0AK9sSWEG9z7QzriE4oQWW5y4iq1cY/O44ivX2DyueIoSl5nltAD/G/jvgHI9J3LuGcC9ke3PAZ/LSHMvcEa4ngC2Z3sv4EFgbi55ZlvmzJljuViyZElO6czMbn3kBTvis7+xra/35HxOvuLEVWzlGpvHFV+5xuZxxTOacQFLLct3apzmrFnAx4GXJd0t6UuS3pfjuU8Ax0iaKSkJXAgszkizGLgkXF8IPGhmFp6TAJB0BHAc8EKOeRZF8945Rfy+iHOuusSZT+S9AJIaCAqUNwKnkUPTlpn1SbqcoLZRC/zQzFZLuoagdFsM3AzcKmkd8BpBoQAwD7hSUi/BTIqfMLPtYSwH5Jnr5xlNjUmf3dA5V53i3BMBwMx6gGXhEue8e4B7MvZdFVnfDRxQszGzW4Fbc82zFHxiKudctfIn1kfB3ompfIpc51yV8UJkFDTVpyem8nsizrnqklMhosC0QgdTqdI1Eb8n4pyrNjkVImH3rpLfeyhXXog456pVnOasJyWdUrBIKlhTMl2IeHOWc666xOmddRrwIUkvAF3sm5TqxEIEVklqa8S4uhq6/Ma6c67KxClE3lmwKMYAH8nXOVeN4jRnbQLeAlxiZhsJRtNtK0hUFaipPkG3FyLOuSoTpxD5DsF4VReF27sI5vNwQGMy4V18nXNVJ9Y9ETN7k6TlAGbWEY5Z5YDmep/d0DlXfeLURHrD2QQNQNLBBGNZOYLmLL+x7pyrNnEKkW8BdwFTJH0JeBj4ckGiqkBN9QmviTjnqk6cUXxvk7QMOJuge+8FZramYJFVmKZkrT8n4pyrOjkXIpKuM7PPAs9m2Vf1vCbinKtGcZqzzs2y77zRCqTSNYf3RMznWXfOVZFhayKSPg58AjhS0tORQ+OBPxUqsErTmEwwYLC7d4CGcJIq55wb63JpznoX8LfAWuDdkf27zOy1gkRVgZr3Dgff54WIcw6AHd0pNmzvYmdPLy0NdcxsbWJi49h6MiKX5qyjgF6CQmQnwUOGuwAkHVS40CqLj+TrnIva0Z1i+aYOUn0DTGpMkuobYPmmDnZ0p0od2qjKpSbyPeABYCbBlLiKHDPgyALEVXH2FiL+rIhzDtiwvYvGZILGcJTv9OuG7V3Mnj52aiPD1kTM7FtmdjzwH2Z2pJnNjCxegIR8OHjnXNTOnl4a6vZv2m6oq2VnT2+JIiqMnHtnmdnHJU2SdKqkt6aXXM+XNF/SWknrJF2Z5Xi9pNvD449JmhHuP1fSMkkrw9e3R85pD/NcES5Tco1ntKWnyPXmLOccQEtDHT29+/9T2dPbT0tDXYkiKow4z4l8DPg0MBVYAZwOPAK8fajzwnNrCQZrPBfYDDwhabGZPRNJdinQYWZHS7oQuA54P7AdeLeZvSTpBOBe4PDIeR80s6W5fo5CafbmLOdcxMzWJpZv6gCCGkhPbz/dqT6OPWRSiSMbXXGeE/k0cAqw0czOAmYDO3I891RgnZmtN7MUsAhYkJFmAXBLuH4ncLYkmdlyM3sp3L8aaJBUHyPuovAb6865qImNSWZPn0QyUUNHd4pkoobZ0yeNud5ZyvXhOElPmNkpklYQjOi7R9JqM3tDDucuBOab2cfC7YvDPC6PpFkVptkcbj8fptmekc9/M7Nzwu12YDLQD/wcuNayfCBJlwGXAbS1tc1ZtGjRsJ+3s7OT5ubmYdOldfUan3ygm4uOS/LOGYWrrsaNq5jKNTaPK75yjc3jimc04zrrrLOWmdncAw6YWU4LweCLE4GrgT8AvwLuyfHchcBNke2LgW9npFkFTI1sPw+0RrbfEO47KrLv8PB1PPB74MPDxTJnzhzLxZIlS3JKl9bb129HfPY39s37n4t1Xlxx4yqmco3N44qvXGPzuOIZzbiApZblOzXOAIzvCVevlrQEmAD8LsfTtwDTIttTw33Z0myWlAjzfxVA0lSCQuzDZvZ8JKYt4esuST8laDb7ca6faTQlamuoT/g868656hLnnsheZvaQmS224P5GLp4AjpE0M5zI6kJgcUaaxcAl4fpC4EEzM0kTgbuBK81s7zArkhKSWsP1OoKn6lfl83lGiw/C6JyrNnkVInGZWR9wOUHPqjXAHWa2WtI1ks4Pk90MTJa0DrgCSHcDvhw4GrgqoytvPXBvOJ7XCoKazA+K8XkG01Tvw8E756pLnOlxR8TM7gHuydh3VWR9N/C+LOddC1w7SLZzRjPGkWpKek3EOVddYtdEJDWFz324DM0+Ra5zrsoMW4hIqpH0AUl3S9pGMCnVy5KekfRVSUcXPszK0FifoNObs5xzVSSXmsgSgpF8PwccYmbTzGwKMA94FLhO0ocKGGPFaK6v9eYs51xVyeWeyDlmdsCIYRbMJfJz4Odh76iq15RM0O2FiHOuiuQyim8vgKRvStJQaapdU32CTi9EnHNVJM6N9V3AYklNAJLeKcmnx41oqq+lK9Xv86w756pGnCfWvyDpA0C7pBTQyb5nORxBTaR/wNjTN8C4Ou/A5pwb++IMBX828I9AF3Ao8FEzW1uowCpRc2QkXy9EnHPVIE5z1r8CXzSzMwmGJbk9OkGU2zf9pT+17pyrFnGas94eWV8p6TyC3llvLkRglag5nN3Qb64756pFLg8bDtYj62Xg7KHSVJv0xFTd/tS6c65K5PSwoaR/kjQ9ujMcjfcMSbewb/TdqpZuzvKaiHOuWuTSnDUf+CjwM0kzCabEHQfUEkwE9Q0zW164ECvHvhvrfk/EOVcdcilErjOzT0v6EdALtAI9Zpbr/OpVoym8J+JDnzjnqkUuzVlvDV//aGa9ZvayFyDZ7a2J+D0R51yVyKUQeUDSI8Ahkj4qaY6k+kIHVon2dfH1QsQ5Vx2Gbc4ys89IOopgNN+ZwPnAG8Kn1leZ2fsLHGPFSCZqSNbW+HDwzrmqkdNzImb2vKRzzOy59D5JzcAJBYusQjXV13oXX+dc1YgzPe7GcOysGRnnPTqqEVW4xqSP5Oucqx5xhj35FbAA6CMYPyu95ETSfElrJa2TdMDAjZLqJd0eHn9M0oxw/7mSlklaGb6+PXLOnHD/OknfKoeHHpvrfZ5151z1iFMTmWpm8/N5k3BO9huAc4HNwBOSFpvZM5FklwIdZna0pAuB64D3A9uBd5vZS5JOAO4FDg/P+S7BoJCPAfcQPNPy23xiHC1N9bX+nIhzrmrEqYn8WdIb83yfU4F1ZrbezFLAIoJaTdQC4JZw/U7gbEkys+Vm9lK4fzXQENZaDgVazOxRCybw+DFwQZ7xjZqm+oR38XXOVY04NZF5wEckbQD2AALMzE7M4dzDgf1EyjYAABYuSURBVBcj25uB0wZLY2Z9kl4HJhPURNLeCzxpZnskHR7mE83zcLKQdBlwGUBbWxvt7e3DBtzZ2ZlTukzdO3ezrWsgr3NzkW9cxVCusXlc8WWLrX/ASPUN0G9GrUQyUUNtTXFbkMv1mlVzXHEKkfMKFkUOJL2BoInrHXHPNbMbgRsB5s6da2eeeeaw57S3t5NLuky/3vYUr6x/Na9zc5FvXMVQrrF5XPFlxrajO8XyTR0clEzQUFdLT28/3ak+Tp4+iYmNyZLFVS6qOa44Q8FvHMH7bAGmRbanhvuypdksKQFMAF4FkDQVuAv4sJk9H0k/dZg8i665vtabs9yYs2F7F43JxN4HatOvG7Z3MXt68QoRV35yGQr+4fB1l6Sd4Wt62Znj+zwBHCNpZjj674XA4ow0i9k3GvBC4EEzM0kTgbuBK81s75zu4VD0OyWdHvbK+jBBD7KSaixR76z0f4oPrd3G8k0d7OhOFT0GN3bt7OmlIWO2zoa6Wnb29JYoIlcuhi1EzGxe+DrezFrC1/TSksubmFkfcDlBz6o1wB1mtlrSNZLOD5PdDEyWtA64gn3zt18OHA1cJWlFuEwJj30CuAlYBzxPiXtmQdDFt7ff2NNXvB5a6QIk1TfApMYkqb4BL0jcqGppqKOnd//f6Z7efloa6koUkSsXceZYnwt8noyHDXO8sY6Z3UPQDTe676rI+m7gfVnOuxa4dpA8l1JmT803JdMj+fZTnyjOPOve1OAKbWZrE8s3dQDsd0/k2EMmlTgyV2pxbqzfBvwLsBIYKEw4lS/9vOODa7Zy1JRmZrY2FfzG486eXiZlvEdDXS0dkZrIju4UG7Z3sbOnl5aGuqLE5fJTjj+riY1JZk+fxIbtXXR0p2hpqOPYQ4p7U92VpziFyF/NLPM+hovY0Z1i684eIBiMMd2sNLvAPVjSTQ3pGgjs39SQbu5qTCaY1Jikp7e/KHG5+Mr5ZxUUJP774vYXpxD5N0k3AQ8QPCcCgJn9YtSjqlAbtncxoSH4I7t75Ssc3Jwk1T/Afc9s5dAJDaPzHi+kWNn/l/327e7t56XXe6irraGutobe/gF6+wc4bEIDD/9lOy+/3kPfgJGs3XcLbLTjGiy2clBJceX6s0omavjg6UfsncPGuVKJ8xv4D8BxQB37mrMM8EIktLOnlxmTG6lP1HDv6lcK90Z/eW74NKVSrrGNwbgStTVcOm/mKAbjXHxxCpFTzOzYgkUyBrQ01DGurpbbLztj777uVB/JRA0nT5s4Ku/x0EMP8ba3vS3WOSte3EGqb2C/5q7Rjivf2IqhkuLK9Wd13jf/yANrtnoh4kouTiHyZ0mzMgZNdBHZerDs6etn1mEtJGrjDFM2uNoaxc7r6CnNLN/UQW2NChZXvrEVQyXFlevP6pxZbfzgD+t5vaeXCd7N1pVQnL+s04EV4XDuT4dDsD9dqMAqUboHSzJRQ0d3imSipoxuiJZfXO5Auf6szjm+jb4B46Hn/lqiSJ0LxKmJ5DUMfLUp1x4s5RqXO1AuP6uTp02ktTnJ/c9s5fyTDitSZJWpHLtM5xJXucadKeeaiJltzLYUMjjnXHa1NeLtx01hydpt9Pb7Y1uDKdfRHIaLq1zjzqb8Goqdczk55/g2du3u44kNr5U6lLIVHc1B0t71DdtznpS1JHGVa9zZeCHiXIWad0wryUQN963ZWupQyla5Dhw5XFzlGnc2Xog4V6EakwnmHd3K/Wu2Ekzu6TKV68CRw8VVrnFn44WIcxXsnOPbePG1Hp7b2lnqUMrSzNYmulN9dKf6MLO96zNbm8o6rnKNOxsvRJyrYGcfH8yKcH8ZNGmV45w25dq9fbi4yjXubHzgHecqWFvLOE6aOoH712zlk2cdXbI4fODI+IaLq1zjzuQ1Eecq3DnHt7HixR1s27W7ZDFUUm8iN7q8EHGuwp0zqw0zWPLstpLFUEm9idzo8kLEuQp33CHjOXxiA/c9U7pCpJJ6E7nR5YWIcxVOEuccP4WH1/2VnlT/8CcUQCX1JnKjq2iFiKT54eCN6yRdmeV4vaTbw+OPSZoR7p8saYmkTknfzjinPcxzRbhMKc6nca68nDOrjd29A/xp3faSvH8l9SZyo6sovbMk1QI3AOcCm4EnJC3OGFb+UqDDzI6WdCFwHfB+YDfwReCEcMn0QTNbWtAP4FyZO23mZJrrE9y/ZivnzGorSQyV0pvIja5i1UROBdaZ2XozSwGLgAUZaRYAt4TrdwJnS5KZdZnZwwSFiXMui2SihrcdezD3r9nGwIA/ve6KR8UYLkHSQmC+mX0s3L4YOM3MLo+kWRWm2RxuPx+m2R5ufwSYm3FOOzAZ6Ad+DlxrWT6QpMuAywDa2trmLFq0aNiYOzs7aW5uzuvzFlK5xgXlG1u1xPXnl/q48ek9XHX6OI6cWDv8CUOolms2WqohrrPOOmuZmc094ICZFXwBFgI3RbYvBr6dkWYVMDWy/TzQGtn+SJZzDg9fxwO/Bz48XCxz5syxXCxZsiSndMVWrnGZlW9s1RJXR9ceO/Jzd9tXf/fsiPOqlms2WqohLmCpZflOLVZz1hZgWmR7argvaxpJCWAC8OpQmZrZlvB1F/BTgmYz56rSxMYkp8yYVBZDoLjqUaxC5AngGEkzJSWBC4HFGWkWA5eE6wuBB8PSLytJCUmt4Xod8LcEtRnnqtbfHN3Ks6/s4s6lL2Ydv6pU41uV47haY1n6enfu6Sv49S5KIWJmfcDlwL3AGuAOM1st6RpJ54fJbgYmS1oHXAHs7QYs6QXg68BHJG2WNAuoB+4N53lfQVCT+UExPo9z5WhHd4qDm4PeUWtf2VU2s+VV0ix9Y0H0etfWqODXu2gDMJrZPcA9GfuuiqzvBt43yLkzBsl2zmjF51yl27C9i5mtzUyb1ED7c39lcnM9qb5+1ry8i5mtTWzY3kVf/wDJxL6b7tHjUate6aNn5cujFleu7zulZRxvmj4RSaPy3tUoOo7ZawTzzqT3F6ILto/i69wYsbOnl0mNSd58dCu3P/Ei//67Z0eW4YonRyewmI6Z0swHT5vO382ZSss4HzYlrvTvQVRDXS0dlV4Tcc4VVnr8qotOmc5bjm7FDHb39VNXK447tIVnX95Jb78xLlIjiB6PeuKJpZxyyoG9OfMR532f3vw6tz26kat//QzX/W4tF8w+jA+edgRTJzWwYXvX3jb+ma1N/jT8INK/B+kaCBR2HDMvRJwbI2a2NrF8UwcA0w9qpKe3n+5U397hRw5pGbd3zo+GutoDjke9Mr6G4w5pyfY2scV53+MOaeHv505j5ebX+cmjG7lr+RZ+9viLHNnaxPwTDmFSt9G9dRcrN+/guENbGF+kmkpDXS1HTG6siGa26O8BsHccs2MPmVSQ9/NCxLkxIj1+1YbtXXR0p2hpqOPYQw6cLW+w46WKK5s3Tp3AdQtP5PPvOp5vPvAcv1v9Ct9pfz44+PjKgsY7mBMOb+FDpx3B+Scftt9/+eUmer37B4xkoqagP+fyvRLOudjKdba8fN93QmMdb/svB7PgpMNYu7WTdc+tYfLUozCDrj29vOHwCQWI9kAvv76bRY+/yJW/WMmX7lnDe980lQ+dPp3W5vqybGZLX+/X1yeYPb0wNZA0L0Scc2WtpaGO3X0DHH9oC+N31DL1qFa6U317Rwoulo+8eQZLN3Zw6yMbue2xjfzozy9w3CHjmf+GQzhiADp39/HY+lc5adrEUS1I6hM1Zd2M5oWIc66sFbuNfzCSOGXGQZwy4yC2d87iG/c/x72rt/KNB/4SJHjo8YK876xDW/j+xXOYdlBjQfIfKS9EnHNlrdht/Lloba7n3OPbeO/sqax4cQdPrXmOCVOmAkZPqp+ZB4/OoIe9/QPc/PAGzv/2w3zng3M446jJo5LvaPJCxDlX9orZxp+rloY6Un0DzJ1xEIf0JJh6/NSCNLO9+6TD+McfL+Ximx/j6vPfwIdOP2LU8h4NPj2uc87lITolMFCwKYFntjbxi0+8mbcc08oXfrmKL/xyJb39A6P6HiPhNRHnnMtDMZvZWsbVcdMlp/CVe5/l+w+tZ922Tr78d29kR3cvO3t6aWmoK1nPMK+JOOdcntIFSXN9ouBzytfWiM+ddzzXv/8knty4g7//3iOs29ZZ8kEtvSbinHMV5D2zp5LqG+Dff/ssX/zVKo6ZMh6A/gGjRtA8bt/X+o4dPZx8aqqghZvXRJxzrsIc0jKOr7/vJOYecRADZgyYIUFvvzFg7F2KMPu510Scc67SpHuGfXb+cXv3ZesZ1t7eXvD7JF4Tcc65ChPtGWZmBesZlgsvRJxzrsKkb+gnEzV0dKf21kBK0TvLm7Occ64ClWowzUxeE3HOOZe3ohUikuZLWitpnaQrsxyvl3R7ePwxSTPC/ZMlLZHUKenbGefMkbQyPOdbKuehLp1zbgwqSiEiqRa4ATgPmAVcJGlWRrJLgQ4zOxq4Hrgu3L8b+CLwmSxZfxf4R+CYcJk/+tE755wbTLFqIqcC68xsvZmlgEXAgow0C4BbwvU7gbMlycy6zOxhgsJkL0mHAi1m9qiZGfBj4IKCfgrnnHP7KVYhcjjwYmR7c7gvaxoz6wNeB4Ya9/jwMJ+h8nTOOVdAVdE7S9JlwGUAbW1ttLe3D3tOZ2dnTumKrVzjgvKNzeOKr1xj87jiKUZcxSpEtgDTIttTw33Z0myWlAAmAK8Ok+fUYfIEwMxuBG4EmDt3rp155pnDBtze3k4u6YqtXOOC8o3N44qvXGPzuOIpRlzFas56AjhG0kxJSeBCYHFGmsXAJeH6QuDB8F5HVmb2MrBT0ulhr6wPA78a/dCdc84NRkN8T4/uG0nvAr4B1AI/NLMvSboGWGpmiyWNA24FZgOvARea2frw3BeAFiAJ7ADeYWbPSJoL/AhoAH4L/NNQBU+Y11+BjTmE3Apsj/1BC69c44Lyjc3jiq9cY/O44hnNuI4ws4MzdxatEKk0kpaa2dxSx5GpXOOC8o3N44qvXGPzuOIpRlz+xLpzzrm8eSHinHMub16IDO7GUgcwiHKNC8o3No8rvnKNzeOKp+Bx+T0R55xzefOaiHPOubx5IeKccy5vXohkMdyw9aUi6YVw6PsVkpaWOJYfStomaVVk30GS7pP0l/B10lB5FDGuqyVtCa/bivCZpWLHNS2c0uAZSaslfTrcX9JrNkRcJb1mksZJelzSU2Fc/yvcPzOcKmJdOHVEUWdlGiKuH0naELleJxczrowYayUtl/SbcLuw18zMfIksBA9DPg8cSfBw41PArFLHFcb2AtBa6jjCWN4KvAlYFdn3FeDKcP1K4Loyietq4DMlvl6HAm8K18cDzxFMi1DSazZEXCW9ZoCA5nC9DngMOB24g+BBZIDvAR8vk7h+BCws5e9YJMYrgJ8Cvwm3C3rNvCZyoFyGra96ZvYHgpEFoqLD+d9CCYbmHySukjOzl83syXB9F7CGYNTpkl6zIeIqKQt0hpt14WLA2wmmioDSXK/B4ioLkqYC/xW4KdwWBb5mXogcKJdh60vFgN9LWhaOTFxu2iwY0wzgFaCtlMFkuFzS02FzV9Gb2aLCWTtnE/wXWzbXLCMuKPE1C5tlVgDbgPsIWgh2WDBVBJTobzMzLjNLX68vhdfrekn1xY4r9A3gfwID4fZkCnzNvBCpLPPM7E0EM0R+UtJbSx3QYCyoO5fLf2jfBY4CTgZeBr5WqkAkNQM/B/7ZzHZGj5XymmWJq+TXzMz6zexkghG6TwWOK3YM2WTGJekE4HME8Z0CHAR8tthxSfpbYJuZLSvm+3ohcqBchq0vCTPbEr5uA+4i+MMqJ1vDGSfTM09uK3E8AJjZ1vAPfwD4ASW6bpLqCL6obzOzX4S7S37NssVVLtcsjGUHsAQ4A5gYThUBJf7bjMQ1P2wWNDPbA/wHpblefwOcHw5Yu4igGeubFPiaeSFyoFyGrS86SU2SxqfXgXcAq4Y+q+iiw/lfQpkMzZ/+kg69hxJct7Bt+mZgjZl9PXKopNdssLhKfc0kHSxpYrjeAJxLcL9mCcFUEVCa65Utrmcj/wiI4J5D0X/HzOxzZjbVzGYQfG89aGYfpNDXrNQ9CcpxAd5F0EvleeBfSx1PGNORBD3FngJWlzou4GcEzRy9BO2slxK0vz4A/AW4HzioTOK6FVgJPE3wpX1oCeKaR9BU9TSwIlzeVeprNkRcJb1mwInA8vD9VwFXhfuPBB4H1gH/CdSXSVwPhtdrFfATwh5cpVqAM9nXO6ug18yHPXHOOZc3b85yzjmXNy9EnHPO5c0LEeecc3nzQsQ551zevBBxzjmXNy9EnHPO5c0LEeecc3nzQsRVFEkm6WuR7c9IunoU8p0RnYOkkCR9StIaSbdlOfZnSRMlfWKU3/OAPCX9eTTfw1UnL0RcpdkD/J2k1lIHEqVArn9PnwDOtWBIiv2Y2ZuBiWGa0YzhgDzD93JuRLwQcZWmD7gR+O/RnZk1iXQNJdz/bDjz3HOSbpN0jqQ/KZhNMDpQXiI8vkbSnZIaw7w+FM5mt0LS9yXVRt5zraQfEwx3MS0jpiskrQqXfw73fY9gGIrfStrvM4THO4F/B44K3++rcWOQ9MtwuoDVkSkDsuXZOVickbzXSPpBmNfvJTWE47jdrWB2v1WS3p/lc/xC0rWS/iBpk6Rzhvm5ukpVyvFdfPEl7gJ0Ai0EszxOAD5DMAvfDPafzTC6vw94I8E/TcuAHxLMULcA+GWYfgbBGFJ/E27/MMzjeODXQF24/zvAhyPnDACnZ4lzDsFYSk1AM8F4Z7PDYy8wyAyV4efL/CyxYiAcfwtoIChYJmfmGXmvoeJMX7uTw+07gA8B7wV+EMlnQpbP8RfCmREJBnD8j1L/7vhSmCU9PLBzFcPMdob/eX8K6MnhlA1mthJA0mrgATMzSSsJvijTXjSzP4XrPwnz303wRftEMEArDew/XPtGM3s0y3vOA+4ys67wfX8BvIVg8L64zo4Zw6ckvSdcnwYcQzDhVTbDxbnBzFaE68sIrtcdwNckXUcwyN8foxmGNbgJwPXhrjpgR86f1lUUL0RcpfoG8CTB3A0Q/MccbZ4dF1nfE1kfiGwPsP/fQOZopEZQY7nFzD43SBxdMWLOV84xSDoTOAc4w8y6JbWz/7WIK3rt+oEGM3tO0psIRvu9VtIDZnZNJN0sYJmZ9YfbJ1J+0xa4UeL3RFxFMrPXCP4jvjTctRWYImmygqlJ/zaPbKdLOiNc/wDwMMEw7QslTQGQdJCkI3LI64/ABZIaFcz/8p5wXy52AeMj23FimAB0hAXIccDpg+SZd5ySDgO6zewnwFeBN2UkeSPBkPJpJxIMne7GIC9EXCX7GtAKYGa9wDUE8ybcBzybR35rCaYdXgNMAr5rZs8AXyCY2/7pMO9Dh8iDMJ4ngR+F8TwG3GRmOTVlmdmrwJ/Cm9ZfjRnD7wg6CKwhuJn+aLY8RxjnG4HHFcwz/m/AtVmORwuRE/CayJjl84k455zLm9dEnHPO5c0LEeecc3nzQsQ551zevBBxzjmXNy9EnHPO5c0LEeecc3nzQsQ551ze/j+q56KJywbNdgAAAABJRU5ErkJggg==\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"result.plot_convergence();"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"然后，我们想用`hiplot`绘制高维交互图。我们先整理出数据："
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"data = result.plot_hi(return_data_only=True, target_name=\"accuracy\", loss2target_func=lambda loss: 1 - loss)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"如果直接绘图的话，您会发现由于算法选择的存在，会导致高维交互图非常的杂乱。我们简单编写一个函数，对不同的算法选择结果的超参进行可视化："
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"import hiplot as hip"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"pycharm": {}
},
"outputs": [],
"source": [
"def viz_subset(model_name, data):\n",
" data_filtered = []\n",
" for datum in data:\n",
" if datum[\"classifier:__choice__\"] == model_name:\n",
" datum = datum.copy()\n",
" score = datum.pop(\"accuracy\")\n",
" AS, HP = layering_config(datum)[\"classifier\"].popitem()\n",
" HP[\"accuracy\"] = score\n",
" data_filtered.append(HP)\n",
" hip.Experiment.from_iterable(data_filtered).display()"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们简单地对随机森林 `RandomForestClassifier` 进行可视化："
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"因为绘制`hiplot`高维交互图后`jupyter notebook`的文件大小会大量增加，所以我们以截图代替。您可以在自己的`notebook`中取消注释并执行以下代码："
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [],
"source": [
"#viz_subset(\"RandomForestClassifier\", data)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"#viz_subset(\"LinearSVC\", data)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [],
"source": [
"#viz_subset(\"KNeighborsClassifier\", data)"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"pycharm": {}
},
"source": [
"我们将所有代码整理为了 `05. Implement a Simple AutoML System.py` 脚本。\n",
"\n",
"您可以在这个脚本中更直接地学习一个简单AutoML系统的搭建方法。"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "auto-sklearn",
"language": "python",
"name": "auto-sklearn"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.10"
}
},
"nbformat": 4,
"nbformat_minor": 4
}