心内求法

Holbrook的个人博客

0%

用神经网络识别手写数字

Posted on Edited on In

本文主要参考74行代码实现手写数字识别，
但是没有采用Michael Nielsen的neural-networks-and-deep-learning代码，而是直接使用scikit-learn的MLPClassifier(Multi-layer Perceptron Classifier, 多层感知机分类器)。
数据来自著名的MNIST数据集。

本文的代码需要如下python模块：

  • matplotlib
  • PIL(pillow)
  • numpy
  • scikit-learn
  • scipy
  • Theano

神经网络建模的一般过程

神经网络分析的一般过程为：导入数据，训练模型，优化模型，启发式理解等。如下图:

用神经网络解决数字识别问题的思路就是：

  1. 获取大量的手写数字的图像，并且已知它们表示的是哪个数字
  2. 以此为训练样本集合，自动生成一套神经网络模型
  3. 依靠它来识别新的手写数字

生成模型

生成模型是这样一个逐步确定未知参数的迭代过程：

  1. 选定一个基础模型
  2. 设定初始化参数代入模型
  3. 用训练集对模型进行训练
  4. 通过一些数量指标，评估训练误差
  5. 如果训练误差不满足要求，继续调整参数
  6. 重复3–5

生成的模型只是对训练集表现较好，为了验证模型的有效性，还需要通过与训练集无关的测试集，对模型进行检验。

筛选模型

如果有多个模型，要从中选出最好的，很自然的思路就是让多个模型跑测试集，从中挑出误差最小的。
但这种思路有一个明显的问题：如果把模型编号作为一个参数，上述过程就相当于用测试集进行训练。

如果从多个模型的角度来看，有一些“决定模型”的参数，比如训练次数、梯度下降过程的步长、规范化参数、学习回合数、minibatch 值等等，我们把他们叫做超参数。
超参数是更高层次的，模型框架的参数，可以理解成参数的参数。

为了确定超参数，可以把数据再分出一部分，叫做“交叉验证集”。经过交叉验证的模型，依然不能保证在新的数据继续有效，
所以最后还是需要一个新的测试集对模型进行考核评价。

筛选模型，就是确定“超参数”的过程。

至此，规范的方法的是将数据集拆分成三个集合：训练集、交叉验证集、测试集，
依次训练参数、超参数，最终得到最优的模型。

这是一个反复迭代不断优化的过程。其中很大一部分工作是在调整参数和超参数。

需要注意的是，在调优超参数时，一定要将多个模型用交叉验证集的结果来横向比较，
选出最优模型后再用一个新的测试集来最终评估该模型。
绝对不能直接用测试集进行调优，否则很容易发生过拟合.

问题建模

回到手写数字识别的问题。以三层(单隐层)神经网络为例：

  • 输入层：将每个图片规整到28*28=784个像素。 每个节点可以输入0–255（像素点的灰度）
  • 输出层：0-9这10个数字。每个节点可以输出0或1
  • 中间层：用n个节点进行处理，其中n是超参数

代码

##数据探索

%pylab inline
import gzip
import pickle
from PIL import Image

Populating the interactive namespace from numpy and matplotlib

# Load the MNIST data
# 以mnist为例，inf的类型为元祖tuple，他又包含了三个元祖，分别对应训练集，验证集，测试集。
# 每个元祖中又包含两个numpy.ndarray，分别对应训练数据和label数据。训练数据的组成是由50000个含有784个元素的列表组成，
# 每个列表代表一张图片。label数据集是由50000个元素组成的一维numpy.ndarray向量。

f = gzip.open('./mnist.pkl.gz', 'rb')
training_data, validation_data, test_data = pickle.load(f,encoding='latin1')
f.close()

training_data
Out[4]:
(array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
        ..., 
        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
        [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32),
 array([5, 0, 4, ..., 8, 4, 8], dtype=int64))
#变换成28*28
data = 1024-np.reshape(training_data[0][0],(28,28))*1024
new_im = Image.fromarray(data)
# 显示图片
plt.imshow(new_im,plt.cm.gray)
training_data[1][0]
Out[5]:
5

训练

这里省去了优化参数的过程。

%pylab inline
import gzip
import pickle
from PIL import Image
from sklearn.neural_network import MLPClassifier

Populating the interactive namespace from numpy and matplotlib

# 加载数据
with gzip.open('./mnist.pkl.gz', 'rb') as fp:
    training_data,valid_data,test_data = pickle.load(fp,encoding='latin1')

# 设置神经网络模型参数
mlp = MLPClassifier(solver='sgd', activation='relu',alpha=1e-4,
                    hidden_layer_sizes=(50,50), random_state=1,max_iter=10,verbose=10,learning_rate_init=.1)
x,y = training_data

# 训练模型
mlp.fit(x,y) 

Iteration 1, loss = 0.34478992
Iteration 2, loss = 0.14381284
Iteration 3, loss = 0.10958842
Iteration 4, loss = 0.08854350
Iteration 5, loss = 0.07551019
Iteration 6, loss = 0.06716950
Iteration 7, loss = 0.05534532
Iteration 8, loss = 0.05041668
Iteration 9, loss = 0.04786168
Iteration 10, loss = 0.04008910

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)
Out[3]:
MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,
       beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(50, 50), learning_rate='constant',
       learning_rate_init=0.1, max_iter=10, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       solver='sgd', tol=0.0001, validation_fraction=0.1, verbose=10,
       warm_start=False)
# 查看模型结果
x,y=test_data
mlp.score(x,y),mlp.n_layers_,mlp.n_iter_,mlp.loss_,mlp.out_activation_
Out[4]:
(0.97250000000000003, 4, 10, 0.040089104679913064, 'softmax')
# 保存模型为pickle文件

with open('model.pickle', 'wb') as f:
    pickle.dump(mlp, f)
# 读取模型
with open('model.pickle', 'rb') as f:
    model = pickle.load(f)
model
Out[5]:
MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,
       beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(50, 50), learning_rate='constant',
       learning_rate_init=0.1, max_iter=10, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       solver='sgd', tol=0.0001, validation_fraction=0.1, verbose=10,
       warm_start=False)
# sklearn自带方法joblib
from sklearn.externals import joblib

# 保存模型
joblib.dump(mlp, 'model1.pickle')

#载入模型
model = joblib.load('model1.pickle')
model
Out[8]:
MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,
       beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(50, 50), learning_rate='constant',
       learning_rate_init=0.1, max_iter=10, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       solver='sgd', tol=0.0001, validation_fraction=0.1, verbose=10,
       warm_start=False)
x,y=test_data
y
Out[9]:
array([7, 2, 1, ..., 4, 5, 6], dtype=int64)
model.predict(x)
Out[10]:
array([7, 2, 1, ..., 4, 5, 6], dtype=int64)

交叉验证

用交叉验证工具，可以根据score，优化模型的各个参数。

%pylab inline
import gzip
import pickle
from sklearn import cross_validation
from sklearn.externals import joblib

Populating the interactive namespace from numpy and matplotlib

D:\Anaconda3\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)

# 载入数据
with gzip.open('./mnist.pkl.gz', 'rb') as fp:
    training_data,valid_data,test_data = pickle.load(fp,encoding='latin1')
x,y = valid_data
#载入模型
model = joblib.load('model1.pickle')
# 交叉验证，用准确率(accuracy)作为指标,交叉倍率k-fold 为 10倍

score = cross_validation.cross_val_score(model, x,y , cv=10, scoring='accuracy')

Iteration 1, loss = 0.78187116
Iteration 2, loss = 0.25388510
Iteration 3, loss = 0.16955852
Iteration 4, loss = 0.13240239
Iteration 5, loss = 0.10025403
Iteration 6, loss = 0.07594441
Iteration 7, loss = 0.06047849
Iteration 8, loss = 0.04637888
Iteration 9, loss = 0.03677402
Iteration 10, loss = 0.02490908

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.84187092
Iteration 2, loss = 0.26000448
Iteration 3, loss = 0.18066421
Iteration 4, loss = 0.13851107
Iteration 5, loss = 0.10065400
Iteration 6, loss = 0.07659037
Iteration 7, loss = 0.05990770
Iteration 8, loss = 0.04579279
Iteration 9, loss = 0.03313706
Iteration 10, loss = 0.02731456

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82356902
Iteration 2, loss = 0.24284756
Iteration 3, loss = 0.17153713
Iteration 4, loss = 0.12699891
Iteration 5, loss = 0.09540901
Iteration 6, loss = 0.07154817
Iteration 7, loss = 0.05675038
Iteration 8, loss = 0.04283342
Iteration 9, loss = 0.03224249
Iteration 10, loss = 0.02570411

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.80319192
Iteration 2, loss = 0.25151567
Iteration 3, loss = 0.17903346
Iteration 4, loss = 0.13283187
Iteration 5, loss = 0.10542308
Iteration 6, loss = 0.08350177
Iteration 7, loss = 0.06375461
Iteration 8, loss = 0.04551676
Iteration 9, loss = 0.03702495
Iteration 10, loss = 0.03019770

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82339307
Iteration 2, loss = 0.25696178
Iteration 3, loss = 0.18196441
Iteration 4, loss = 0.13706206
Iteration 5, loss = 0.10864025
Iteration 6, loss = 0.08161445
Iteration 7, loss = 0.06482672
Iteration 8, loss = 0.04893842
Iteration 9, loss = 0.03598496
Iteration 10, loss = 0.02949324

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.81738844
Iteration 2, loss = 0.25368526
Iteration 3, loss = 1.15094211
Iteration 4, loss = 0.25568068
Iteration 5, loss = 0.18752010
Iteration 6, loss = 0.13994125
Iteration 7, loss = 0.12223943
Iteration 8, loss = 0.12960278
Iteration 9, loss = 0.09046564
Iteration 10, loss = 0.07748831
Iteration 1, loss = 0.82438630

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.25874840
Iteration 3, loss = 1.01485889
Iteration 4, loss = 0.23356027
Iteration 5, loss = 0.17288887
Iteration 6, loss = 0.13914482
Iteration 7, loss = 0.11251162
Iteration 8, loss = 0.10111344
Iteration 9, loss = 0.08639655
Iteration 10, loss = 0.07165107
Iteration 1, loss = 0.81457274

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.25901173
Iteration 3, loss = 0.78948227
Iteration 4, loss = 0.22246483
Iteration 5, loss = 0.16539281
Iteration 6, loss = 0.13721071
Iteration 7, loss = 0.79746186
Iteration 8, loss = 0.25221474
Iteration 9, loss = 0.39058177
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.83040680
Iteration 2, loss = 0.26367480
Iteration 3, loss = 1.00381224
Iteration 4, loss = 0.25186480
Iteration 5, loss = 0.19493333
Iteration 6, loss = 0.15539808
Iteration 7, loss = 0.66227258
Iteration 8, loss = 0.23923389
Iteration 9, loss = 0.52421580
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.83035768
Iteration 2, loss = 0.25720301
Iteration 3, loss = 0.82076672
Iteration 4, loss = 0.23261915
Iteration 5, loss = 0.15946379
Iteration 6, loss = 0.12189619
Iteration 7, loss = 0.18415185
Iteration 8, loss = 0.09925235
Iteration 9, loss = 0.11875773
Iteration 10, loss = 0.06922819

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

# 得到一个scores的预测准确率数组
score
Out[5]:
array([ 0.94223108,  0.95508982,  0.91317365,  0.95004995,  0.948     ,
        0.92692693,  0.94194194,  0.91583166,  0.9258517 ,  0.95687061])
print("Accuracy: %0.2f (+/- %0.2f)" % (score.mean(), score.std() * 2))

Accuracy: 0.94 (+/- 0.03)

# 优化训练次数
from sklearn.learning_curve import learning_curve
train_sizes, train_loss, test_loss = learning_curve(model, 
                                                    x, y, cv = 10, scoring = 'neg_mean_squared_error',
                                                    train_sizes = [0.1, 0.25, 0.5, 0.75, 1])

# 训练误差均值
train_loss_mean = -np.mean(train_loss, axis = 1)
# 测试误差均值
test_loss_mean = -np.mean(test_loss, axis = 1)

Iteration 1, loss = 2.16234521
Iteration 2, loss = 1.39525089

D:\Anaconda3\lib\site-packages\sklearn\learning_curve.py:23: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the functions are moved. This module will be removed in 0.20
  DeprecationWarning)

Iteration 3, loss = 0.76439142
Iteration 4, loss = 0.45542935
Iteration 5, loss = 0.28384867
Iteration 6, loss = 0.22150051
Iteration 7, loss = 0.16080946
Iteration 8, loss = 0.09847794
Iteration 9, loss = 0.07672188
Iteration 10, loss = 0.05843624
Iteration 1, loss = 1.75060450
Iteration 2, loss = 0.69644604

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 3, loss = 0.39784156
Iteration 4, loss = 0.29058588
Iteration 5, loss = 0.21158317
Iteration 6, loss = 0.16391721
Iteration 7, loss = 0.12268283
Iteration 8, loss = 0.09531558
Iteration 9, loss = 0.07535034
Iteration 10, loss = 0.06059414

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.27510630
Iteration 2, loss = 0.38422172
Iteration 3, loss = 0.26058371
Iteration 4, loss = 0.19015472
Iteration 5, loss = 0.14916225
Iteration 6, loss = 0.11217251
Iteration 7, loss = 0.08964414
Iteration 8, loss = 0.06752874
Iteration 9, loss = 0.05173431
Iteration 10, loss = 0.03548834

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.00107727
Iteration 2, loss = 0.30418627
Iteration 3, loss = 0.22261019
Iteration 4, loss = 0.16506052
Iteration 5, loss = 0.12553921
Iteration 6, loss = 0.09968461
Iteration 7, loss = 0.07301909
Iteration 8, loss = 0.05305705
Iteration 9, loss = 0.04380569
Iteration 10, loss = 0.03318207

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.78187116
Iteration 2, loss = 0.25388510
Iteration 3, loss = 0.16955852
Iteration 4, loss = 0.13240239
Iteration 5, loss = 0.10025403
Iteration 6, loss = 0.07594441
Iteration 7, loss = 0.06047849
Iteration 8, loss = 0.04637888
Iteration 9, loss = 0.03677402
Iteration 10, loss = 0.02490908

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.75849645
Iteration 2, loss = 0.83352538
Iteration 3, loss = 0.43310897

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 4, loss = 0.31902247
Iteration 5, loss = 0.24905543
Iteration 6, loss = 0.20788410
Iteration 7, loss = 0.15775448
Iteration 8, loss = 0.12920742
Iteration 9, loss = 0.09537131
Iteration 10, loss = 0.07162782

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.24904164
Iteration 2, loss = 0.44540982
Iteration 3, loss = 0.28849654
Iteration 4, loss = 0.21581374
Iteration 5, loss = 0.17587408
Iteration 6, loss = 0.13433313
Iteration 7, loss = 0.10394670
Iteration 8, loss = 0.07836265
Iteration 9, loss = 0.06137090
Iteration 10, loss = 0.04278865

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.98434583
Iteration 2, loss = 0.33796018
Iteration 3, loss = 0.22436210
Iteration 4, loss = 0.17315177
Iteration 5, loss = 0.13051203
Iteration 6, loss = 0.10374355
Iteration 7, loss = 0.07904730
Iteration 8, loss = 0.05511370
Iteration 9, loss = 0.04291312
Iteration 10, loss = 0.02867291

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82400335
Iteration 2, loss = 0.25945869
Iteration 3, loss = 0.18018568
Iteration 4, loss = 0.13052835
Iteration 5, loss = 0.09828557
Iteration 6, loss = 0.07558895
Iteration 7, loss = 0.06001946
Iteration 8, loss = 0.04326780
Iteration 9, loss = 0.03497892
Iteration 10, loss = 0.02263775
Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.73762422
Iteration 2, loss = 0.71349080

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 3, loss = 0.36731060
Iteration 4, loss = 0.24303032
Iteration 5, loss = 0.19761464
Iteration 6, loss = 0.14383139
Iteration 7, loss = 0.11151771
Iteration 8, loss = 0.08367673
Iteration 9, loss = 0.06385257
Iteration 10, loss = 0.04599619

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.32015122
Iteration 2, loss = 0.39880178
Iteration 3, loss = 0.26532128
Iteration 4, loss = 0.19661938
Iteration 5, loss = 0.15439904
Iteration 6, loss = 0.11932913
Iteration 7, loss = 0.09234866
Iteration 8, loss = 0.06945134
Iteration 9, loss = 0.05181438
Iteration 10, loss = 0.03889063

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.97007927
Iteration 2, loss = 0.29845584
Iteration 3, loss = 0.20254406
Iteration 4, loss = 0.15491243
Iteration 5, loss = 0.11462142
Iteration 6, loss = 0.09078088
Iteration 7, loss = 0.07017862
Iteration 8, loss = 0.05102731
Iteration 9, loss = 0.03907040
Iteration 10, loss = 0.02937477

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.83109361
Iteration 2, loss = 0.24353905
Iteration 3, loss = 0.16525866
Iteration 4, loss = 0.11980748
Iteration 5, loss = 0.08934436
Iteration 6, loss = 0.07100511
Iteration 7, loss = 0.05496481
Iteration 8, loss = 0.04479567
Iteration 9, loss = 0.03132412
Iteration 10, loss = 0.02639528
Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 3, loss = 0.37559278
Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.37230731
Iteration 2, loss = 0.43416639
Iteration 3, loss = 0.27931529
Iteration 4, loss = 0.20204367
Iteration 5, loss = 0.15113129
Iteration 6, loss = 0.12601466
Iteration 7, loss = 0.08811592
Iteration 8, loss = 0.06920111
Iteration 9, loss = 0.04975573
Iteration 10, loss = 0.03743508
Iteration 1, loss = 1.01156174

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.31302025
Iteration 3, loss = 0.21761332
Iteration 4, loss = 0.16898564
Iteration 5, loss = 0.12392184
Iteration 6, loss = 0.09119992
Iteration 7, loss = 0.07497565
Iteration 8, loss = 0.05517513
Iteration 9, loss = 0.03923757
Iteration 10, loss = 0.03047696

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.81210365
Iteration 2, loss = 0.25181047
Iteration 3, loss = 0.18210693
Iteration 4, loss = 0.13449020
Iteration 5, loss = 0.10105392
Iteration 6, loss = 0.07959972
Iteration 7, loss = 0.06183630
Iteration 8, loss = 0.04501858
Iteration 9, loss = 0.03330106
Iteration 10, loss = 0.02616531
Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689
Iteration 1, loss = 1.40455534

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.44711574
Iteration 3, loss = 0.29097536
Iteration 4, loss = 0.21783465
Iteration 5, loss = 0.16918544
Iteration 6, loss = 0.13507056
Iteration 7, loss = 0.10530137
Iteration 8, loss = 0.08116109
Iteration 9, loss = 0.05859452
Iteration 10, loss = 0.04298089
Iteration 1, loss = 0.99209277

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.30908262
Iteration 3, loss = 0.22304977
Iteration 4, loss = 0.16646772
Iteration 5, loss = 0.13228428
Iteration 6, loss = 0.10005274
Iteration 7, loss = 0.07785912
Iteration 8, loss = 0.05641814
Iteration 9, loss = 0.04254776
Iteration 10, loss = 0.03414474

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.84388829
Iteration 2, loss = 0.25699326
Iteration 3, loss = 0.17917552
Iteration 4, loss = 0.13574808
Iteration 5, loss = 0.10260718
Iteration 6, loss = 0.08196382
Iteration 7, loss = 0.06122960
Iteration 8, loss = 0.04654568
Iteration 9, loss = 0.03599603
Iteration 10, loss = 0.02739421

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278
Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.29896116
Iteration 2, loss = 0.39703487
Iteration 3, loss = 0.26492047
Iteration 4, loss = 0.20348830
Iteration 5, loss = 0.15320242
Iteration 6, loss = 0.11910508
Iteration 7, loss = 0.08946200
Iteration 8, loss = 0.06924719
Iteration 9, loss = 0.04948867
Iteration 10, loss = 0.03593538
Iteration 1, loss = 1.00864956

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.31490112
Iteration 3, loss = 0.22262727
Iteration 4, loss = 0.17291961
Iteration 5, loss = 0.13377426
Iteration 6, loss = 0.10207598
Iteration 7, loss = 0.07711785
Iteration 8, loss = 0.05479024
Iteration 9, loss = 0.03684906
Iteration 10, loss = 0.03072051

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82621784
Iteration 2, loss = 0.25661650
Iteration 3, loss = 0.18073473
Iteration 4, loss = 0.13184292
Iteration 5, loss = 0.10015016
Iteration 6, loss = 0.07649430
Iteration 7, loss = 0.05869316
Iteration 8, loss = 0.04204414
Iteration 9, loss = 0.03382942
Iteration 10, loss = 0.02457005

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278
Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.29896116
Iteration 2, loss = 0.39703487
Iteration 3, loss = 0.26492047
Iteration 4, loss = 0.20348830
Iteration 5, loss = 0.15320242
Iteration 6, loss = 0.11910508
Iteration 7, loss = 0.08946200
Iteration 8, loss = 0.06924719
Iteration 9, loss = 0.04948867
Iteration 10, loss = 0.03593538
Iteration 1, loss = 1.01484990

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.31694362
Iteration 3, loss = 0.21880210
Iteration 4, loss = 0.16722427
Iteration 5, loss = 0.12608930
Iteration 6, loss = 0.09443923
Iteration 7, loss = 0.07173557
Iteration 8, loss = 0.05395281
Iteration 9, loss = 0.03867746
Iteration 10, loss = 0.02948218

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82813587
Iteration 2, loss = 0.25912779
Iteration 3, loss = 0.18338442
Iteration 4, loss = 0.13464529
Iteration 5, loss = 0.10627939
Iteration 6, loss = 0.08030210
Iteration 7, loss = 0.06209368
Iteration 8, loss = 0.04619573
Iteration 9, loss = 0.03767382
Iteration 10, loss = 0.02751743
Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689
Iteration 1, loss = 1.29896116

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.39703487
Iteration 3, loss = 0.26492047
Iteration 4, loss = 0.20348830
Iteration 5, loss = 0.15320242
Iteration 6, loss = 0.11910508
Iteration 7, loss = 0.08946200
Iteration 8, loss = 0.06924719
Iteration 9, loss = 0.04948867
Iteration 10, loss = 0.03593538

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.98646623
Iteration 2, loss = 0.31715128
Iteration 3, loss = 0.22734489
Iteration 4, loss = 0.17786827
Iteration 5, loss = 0.13861779
Iteration 6, loss = 0.10563623
Iteration 7, loss = 0.08386473
Iteration 8, loss = 0.06692934
Iteration 9, loss = 0.05156518
Iteration 10, loss = 0.03704055

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.81433422
Iteration 2, loss = 0.26343081
Iteration 3, loss = 0.18872972
Iteration 4, loss = 0.14357252
Iteration 5, loss = 0.11438251
Iteration 6, loss = 0.08966009
Iteration 7, loss = 0.06421667
Iteration 8, loss = 0.05022774
Iteration 9, loss = 0.04017863
Iteration 10, loss = 0.02846312
Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985
Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689
Iteration 1, loss = 1.29896116

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.39703487
Iteration 3, loss = 0.26492047
Iteration 4, loss = 0.20348830
Iteration 5, loss = 0.15320242
Iteration 6, loss = 0.11910508
Iteration 7, loss = 0.08946200
Iteration 8, loss = 0.06924719
Iteration 9, loss = 0.04948867
Iteration 10, loss = 0.03593538

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.98646623
Iteration 2, loss = 0.31715128
Iteration 3, loss = 0.22734489
Iteration 4, loss = 0.17786827
Iteration 5, loss = 0.13861779
Iteration 6, loss = 0.10563623
Iteration 7, loss = 0.08386473
Iteration 8, loss = 0.06692934
Iteration 9, loss = 0.05156518
Iteration 10, loss = 0.03704055

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82939742
Iteration 2, loss = 0.26731813
Iteration 3, loss = 0.19012672
Iteration 4, loss = 0.14588727
Iteration 5, loss = 0.11056198
Iteration 6, loss = 0.08888431
Iteration 7, loss = 0.06612879
Iteration 8, loss = 0.05136425
Iteration 9, loss = 0.03855565
Iteration 10, loss = 0.02878478

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 2.19753486
Iteration 2, loss = 1.54314408
Iteration 3, loss = 0.98675974
Iteration 4, loss = 0.60762570
Iteration 5, loss = 0.44343678
Iteration 6, loss = 0.32024609
Iteration 7, loss = 0.23976259
Iteration 8, loss = 0.18756953
Iteration 9, loss = 0.14479318
Iteration 10, loss = 0.12453985

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.76236579
Iteration 2, loss = 0.68918246
Iteration 3, loss = 0.37559278
Iteration 4, loss = 0.26626466
Iteration 5, loss = 0.20200529
Iteration 6, loss = 0.16245522
Iteration 7, loss = 0.12048935
Iteration 8, loss = 0.09836368
Iteration 9, loss = 0.07269702
Iteration 10, loss = 0.05698689

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.29896116
Iteration 2, loss = 0.39703487
Iteration 3, loss = 0.26492047
Iteration 4, loss = 0.20348830
Iteration 5, loss = 0.15320242
Iteration 6, loss = 0.11910508
Iteration 7, loss = 0.08946200
Iteration 8, loss = 0.06924719
Iteration 9, loss = 0.04948867
Iteration 10, loss = 0.03593538
Iteration 1, loss = 0.98646623

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.31715128
Iteration 3, loss = 0.22734489
Iteration 4, loss = 0.17786827
Iteration 5, loss = 0.13861779
Iteration 6, loss = 0.10563623
Iteration 7, loss = 0.08386473
Iteration 8, loss = 0.06692934
Iteration 9, loss = 0.05156518
Iteration 10, loss = 0.03704055

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.85377479
Iteration 2, loss = 0.25900606
Iteration 3, loss = 0.18080784
Iteration 4, loss = 0.13487443
Iteration 5, loss = 0.10611096
Iteration 6, loss = 0.08136923
Iteration 7, loss = 0.06168739
Iteration 8, loss = 0.04903990
Iteration 9, loss = 0.03722986
Iteration 10, loss = 0.02726954

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

# 绘制误差曲线
plt.plot(train_sizes, train_loss_mean, 'o-', color = 'r', label = 'Training')
plt.plot(train_sizes, test_loss_mean, 'o-', color = 'g', label = 'Cross-Validation')

plt.xlabel('Training data size')
plt.ylabel('Loss')
plt.legend(loc = 'best')
plt.show()

# 优化节点数
from sklearn.model_selection import validation_curve
train_score, test_score = validation_curve(model, x, y, 'hidden_layer_sizes', [(10, 10),(20,20),(50, 50),(80,80),(100,100)], cv=10, scoring='accuracy', n_jobs=1)

Iteration 1, loss = 1.40452866
Iteration 2, loss = 0.45277100
Iteration 3, loss = 0.33622163
Iteration 4, loss = 0.28234337
Iteration 5, loss = 0.26423751
Iteration 6, loss = 0.24057286
Iteration 7, loss = 0.22555253
Iteration 8, loss = 0.21236895
Iteration 9, loss = 0.20946425
Iteration 10, loss = 0.19595660

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.90678742
Iteration 2, loss = 0.29778804
Iteration 3, loss = 0.22690549
Iteration 4, loss = 0.18558652
Iteration 5, loss = 0.15274193
Iteration 6, loss = 0.14003401
Iteration 7, loss = 0.11722206
Iteration 8, loss = 0.11416346
Iteration 9, loss = 0.09298024
Iteration 10, loss = 0.08794944

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.78187116
Iteration 2, loss = 0.25388510
Iteration 3, loss = 0.16955852
Iteration 4, loss = 0.13240239
Iteration 5, loss = 0.10025403
Iteration 6, loss = 0.07594441
Iteration 7, loss = 0.06047849
Iteration 8, loss = 0.04637888
Iteration 9, loss = 0.03677402
Iteration 10, loss = 0.02490908

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.72488779
Iteration 2, loss = 0.22713011
Iteration 3, loss = 0.15378707
Iteration 4, loss = 0.10674882
Iteration 5, loss = 0.08116687
Iteration 6, loss = 0.05852362
Iteration 7, loss = 0.04129173
Iteration 8, loss = 0.03260524
Iteration 9, loss = 0.02342850
Iteration 10, loss = 0.01683708

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.76085272
Iteration 2, loss = 0.22664155
Iteration 3, loss = 0.14677409
Iteration 4, loss = 0.10715156
Iteration 5, loss = 0.07679921
Iteration 6, loss = 0.06052605
Iteration 7, loss = 0.04205178
Iteration 8, loss = 0.02853183
Iteration 9, loss = 0.01974877
Iteration 10, loss = 0.01493905

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.66344874
Iteration 2, loss = 0.59623043
Iteration 3, loss = 0.38985668
Iteration 4, loss = 0.33769992
Iteration 5, loss = 0.31157361
Iteration 6, loss = 0.28956695
Iteration 7, loss = 0.26731122
Iteration 8, loss = 0.25674761
Iteration 9, loss = 0.24826723
Iteration 10, loss = 0.24057102

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.92889035
Iteration 2, loss = 0.32173874
Iteration 3, loss = 0.24504576
Iteration 4, loss = 0.19791168
Iteration 5, loss = 0.16720739
Iteration 6, loss = 0.14356989
Iteration 7, loss = 0.12254948
Iteration 8, loss = 0.09748972
Iteration 9, loss = 0.08695943
Iteration 10, loss = 0.08548024

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.84187092
Iteration 2, loss = 0.26000448
Iteration 3, loss = 0.18066421
Iteration 4, loss = 0.13851107
Iteration 5, loss = 0.10065400
Iteration 6, loss = 0.07659037
Iteration 7, loss = 0.05990770
Iteration 8, loss = 0.04579279
Iteration 9, loss = 0.03313706
Iteration 10, loss = 0.02731456

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.73333666
Iteration 2, loss = 0.24119006
Iteration 3, loss = 0.16273019
Iteration 4, loss = 0.11844502
Iteration 5, loss = 0.08505964
Iteration 6, loss = 0.06200148
Iteration 7, loss = 0.04741334
Iteration 8, loss = 0.03341776
Iteration 9, loss = 0.02419570
Iteration 10, loss = 0.01491461

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.79289945
Iteration 2, loss = 0.23190988
Iteration 3, loss = 0.15688516
Iteration 4, loss = 0.11199636
Iteration 5, loss = 0.07906910
Iteration 6, loss = 0.05828002
Iteration 7, loss = 0.04153152
Iteration 8, loss = 0.02881667
Iteration 9, loss = 0.02238725
Iteration 10, loss = 0.01461734
Iteration 1, loss = 1.40683995
D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.42051133
Iteration 3, loss = 0.30099208
Iteration 4, loss = 0.25130697
Iteration 5, loss = 0.22058510
Iteration 6, loss = 0.19790935
Iteration 7, loss = 0.17940932
Iteration 8, loss = 0.16544273
Iteration 9, loss = 0.15113267
Iteration 10, loss = 0.14249716

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.88820528
Iteration 2, loss = 0.30517951
Iteration 3, loss = 0.23722754
Iteration 4, loss = 0.19074524
Iteration 5, loss = 0.16270604
Iteration 6, loss = 0.13683423
Iteration 7, loss = 0.11391411
Iteration 8, loss = 0.10307550
Iteration 9, loss = 0.09320134
Iteration 10, loss = 0.08144817

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82356902
Iteration 2, loss = 0.24284756
Iteration 3, loss = 0.17153713
Iteration 4, loss = 0.12699891
Iteration 5, loss = 0.09540901
Iteration 6, loss = 0.07154817
Iteration 7, loss = 0.05675038
Iteration 8, loss = 0.04283342
Iteration 9, loss = 0.03224249
Iteration 10, loss = 0.02570411

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.70825222
Iteration 2, loss = 0.22368053
Iteration 3, loss = 0.15433644
Iteration 4, loss = 0.11186812
Iteration 5, loss = 0.08044857
Iteration 6, loss = 0.06129406
Iteration 7, loss = 0.04731268
Iteration 8, loss = 0.03405814
Iteration 9, loss = 0.02178953
Iteration 10, loss = 0.01414137

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.77913723
Iteration 2, loss = 0.21691988
Iteration 3, loss = 0.14315145
Iteration 4, loss = 0.10545728
Iteration 5, loss = 0.07583673
Iteration 6, loss = 0.05494215
Iteration 7, loss = 0.04174418
Iteration 8, loss = 0.02781030
Iteration 9, loss = 0.02166380
Iteration 10, loss = 0.01622253

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.63803563
Iteration 2, loss = 0.73438182
Iteration 3, loss = 0.51365135
Iteration 4, loss = 0.43580578
Iteration 5, loss = 0.38821443
Iteration 6, loss = 0.36692558
Iteration 7, loss = 0.34847682
Iteration 8, loss = 0.33140684
Iteration 9, loss = 0.31734486
Iteration 10, loss = 0.30204866
Iteration 1, loss = 0.90417363

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.31126133
Iteration 3, loss = 0.23140628
Iteration 4, loss = 0.19059406
Iteration 5, loss = 0.16168762
Iteration 6, loss = 0.13992754
Iteration 7, loss = 0.11937485
Iteration 8, loss = 0.10166499
Iteration 9, loss = 0.09031415
Iteration 10, loss = 0.07931656

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.80319192
Iteration 2, loss = 0.25151567
Iteration 3, loss = 0.17903346
Iteration 4, loss = 0.13283187
Iteration 5, loss = 0.10542308
Iteration 6, loss = 0.08350177
Iteration 7, loss = 0.06375461
Iteration 8, loss = 0.04551676
Iteration 9, loss = 0.03702495
Iteration 10, loss = 0.03019770

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.71206665
Iteration 2, loss = 0.23780152
Iteration 3, loss = 0.16235581
Iteration 4, loss = 0.11504128
Iteration 5, loss = 0.08697463
Iteration 6, loss = 0.06278477
Iteration 7, loss = 0.04779385
Iteration 8, loss = 0.03613638
Iteration 9, loss = 0.02478665
Iteration 10, loss = 0.01900170

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.77486742
Iteration 2, loss = 0.22861313
Iteration 3, loss = 0.15195083
Iteration 4, loss = 0.10390271
Iteration 5, loss = 0.07445138
Iteration 6, loss = 0.05464828
Iteration 7, loss = 0.03963958
Iteration 8, loss = 0.02826555
Iteration 9, loss = 0.01963422
Iteration 10, loss = 0.01423081

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.22886404
Iteration 2, loss = 0.41983871
Iteration 3, loss = 0.33502410
Iteration 4, loss = 0.31532174
Iteration 5, loss = 0.29367001
Iteration 6, loss = 0.27202519
Iteration 7, loss = 0.25718009
Iteration 8, loss = 0.24745844
Iteration 9, loss = 0.24021322
Iteration 10, loss = 0.22303674

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.90073735
Iteration 2, loss = 0.30368284
Iteration 3, loss = 0.22905183
Iteration 4, loss = 0.19904162
Iteration 5, loss = 0.16713843
Iteration 6, loss = 0.14756067
Iteration 7, loss = 0.13165325
Iteration 8, loss = 0.10924095
Iteration 9, loss = 0.09542321
Iteration 10, loss = 0.08600667

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82339307
Iteration 2, loss = 0.25696178
Iteration 3, loss = 0.18196441
Iteration 4, loss = 0.13706206
Iteration 5, loss = 0.10864025
Iteration 6, loss = 0.08161445
Iteration 7, loss = 0.06482672
Iteration 8, loss = 0.04893842
Iteration 9, loss = 0.03598496
Iteration 10, loss = 0.02949324

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.70926585
Iteration 2, loss = 0.23832791
Iteration 3, loss = 0.16472394
Iteration 4, loss = 0.12345021
Iteration 5, loss = 0.08640567
Iteration 6, loss = 0.06579802
Iteration 7, loss = 0.04877549
Iteration 8, loss = 0.03410949
Iteration 9, loss = 0.02247546
Iteration 10, loss = 0.01663680

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.95122552
Iteration 2, loss = 0.25134753
Iteration 3, loss = 0.16330549
Iteration 4, loss = 0.11461080
Iteration 5, loss = 0.08649812
Iteration 6, loss = 0.06130384
Iteration 7, loss = 0.04624714
Iteration 8, loss = 0.02930188
Iteration 9, loss = 0.02031129
Iteration 10, loss = 0.01277362

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.27876831
Iteration 2, loss = 0.43928526
Iteration 3, loss = 0.35670840
Iteration 4, loss = 0.32391566
Iteration 5, loss = 1.73383758
Iteration 6, loss = 0.89736584
Iteration 7, loss = 0.65255214
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.92024625
Iteration 2, loss = 0.31313336
Iteration 3, loss = 0.23633402
Iteration 4, loss = 3.01835401
Iteration 5, loss = 1.26534345
Iteration 6, loss = 1.05279937
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.81738844
Iteration 2, loss = 0.25368526
Iteration 3, loss = 1.15094211
Iteration 4, loss = 0.25568068
Iteration 5, loss = 0.18752010
Iteration 6, loss = 0.13994125
Iteration 7, loss = 0.12223943
Iteration 8, loss = 0.12960278
Iteration 9, loss = 0.09046564
Iteration 10, loss = 0.07748831

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.75749331
Iteration 2, loss = 0.23578315
Iteration 3, loss = 0.16482743
Iteration 4, loss = 0.11913430
Iteration 5, loss = 0.08509478
Iteration 6, loss = 0.06243259
Iteration 7, loss = 0.04201284
Iteration 8, loss = 1.98774094
Iteration 9, loss = 0.41104280
Iteration 10, loss = 0.26313319
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.79818220
Iteration 2, loss = 0.23199767
Iteration 3, loss = 0.15731305
Iteration 4, loss = 0.11473947
Iteration 5, loss = 0.46137991
Iteration 6, loss = 0.12074858
Iteration 7, loss = 0.08167516
Iteration 8, loss = 0.08107943
Iteration 9, loss = 0.10416250
Iteration 10, loss = 0.04364689

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.34381989
Iteration 2, loss = 0.43655878
Iteration 3, loss = 0.36065320
Iteration 4, loss = 0.30914429
Iteration 5, loss = 1.36973335
Iteration 6, loss = 0.61580174
Iteration 7, loss = 0.47984439
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.89935954
Iteration 2, loss = 0.30888950
Iteration 3, loss = 0.27405683
Iteration 4, loss = 0.63739855
Iteration 5, loss = 0.26321170
Iteration 6, loss = 0.23071879
Iteration 7, loss = 0.17878195
Iteration 8, loss = 0.16068991
Iteration 9, loss = 0.14019449
Iteration 10, loss = 0.12435665

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82438630
Iteration 2, loss = 0.25874840
Iteration 3, loss = 1.01485889
Iteration 4, loss = 0.23356027
Iteration 5, loss = 0.17288887
Iteration 6, loss = 0.13914482
Iteration 7, loss = 0.11251162
Iteration 8, loss = 0.10111344
Iteration 9, loss = 0.08639655
Iteration 10, loss = 0.07165107

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.72063738
Iteration 2, loss = 0.23949650
Iteration 3, loss = 0.16812028
Iteration 4, loss = 0.12829145
Iteration 5, loss = 0.08814535
Iteration 6, loss = 0.06528626
Iteration 7, loss = 0.07555339
Iteration 8, loss = 1.64609863
Iteration 9, loss = 0.30794607
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.77447382
Iteration 2, loss = 0.23399938
Iteration 3, loss = 0.16095154
Iteration 4, loss = 0.11999483
Iteration 5, loss = 0.69665373
Iteration 6, loss = 0.16318997
Iteration 7, loss = 0.10635723
Iteration 8, loss = 0.10349656
Iteration 9, loss = 0.06646714
Iteration 10, loss = 0.04581457

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.24684066
Iteration 2, loss = 0.43211011
Iteration 3, loss = 0.99371805
Iteration 4, loss = 0.52614458
Iteration 5, loss = 0.41352138
Iteration 6, loss = 0.36490241
Iteration 7, loss = 0.33829352
Iteration 8, loss = 0.31701234
Iteration 9, loss = 0.31026822
Iteration 10, loss = 0.29555425

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.00310746
Iteration 2, loss = 0.33447577
Iteration 3, loss = 0.24707166
Iteration 4, loss = 0.20059050
Iteration 5, loss = 0.65855097
Iteration 6, loss = 0.23977948
Iteration 7, loss = 0.20177651
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.81457274
Iteration 2, loss = 0.25901173
Iteration 3, loss = 0.78948227
Iteration 4, loss = 0.22246483
Iteration 5, loss = 0.16539281
Iteration 6, loss = 0.13721071
Iteration 7, loss = 0.79746186
Iteration 8, loss = 0.25221474
Iteration 9, loss = 0.39058177
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.70559083
Iteration 2, loss = 0.23624995
Iteration 3, loss = 0.16647182
Iteration 4, loss = 0.12692616
Iteration 5, loss = 0.09495785
Iteration 6, loss = 0.33746266
Iteration 7, loss = 0.09506544
Iteration 8, loss = 0.06951859
Iteration 9, loss = 0.04492136
Iteration 10, loss = 0.03063069

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.79262790
Iteration 2, loss = 0.23369143
Iteration 3, loss = 0.15470480
Iteration 4, loss = 0.10922491
Iteration 5, loss = 0.08302476
Iteration 6, loss = 0.05751253
Iteration 7, loss = 0.04226786
Iteration 8, loss = 0.03026411
Iteration 9, loss = 0.02176469
Iteration 10, loss = 0.55747035

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 1.32403536
Iteration 2, loss = 0.46850244
Iteration 3, loss = 1.81732645
Iteration 4, loss = 1.09010703
Iteration 5, loss = 1.00059166
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.92601461
Iteration 2, loss = 0.31435463
Iteration 3, loss = 1.06075159
Iteration 4, loss = 0.41228545
Iteration 5, loss = 0.38411835
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.83040680
Iteration 2, loss = 0.26367480
Iteration 3, loss = 1.00381224
Iteration 4, loss = 0.25186480
Iteration 5, loss = 0.19493333
Iteration 6, loss = 0.15539808
Iteration 7, loss = 0.66227258
Iteration 8, loss = 0.23923389
Iteration 9, loss = 0.52421580
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.74428212
Iteration 2, loss = 0.24079066
Iteration 3, loss = 0.16852158
Iteration 4, loss = 0.13372222
Iteration 5, loss = 0.10431823
Iteration 6, loss = 0.08613349
Iteration 7, loss = 0.05470023
Iteration 8, loss = 0.04020246
Iteration 9, loss = 0.02692555
Iteration 10, loss = 0.02730752

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.82783047
Iteration 2, loss = 0.57272682
Iteration 3, loss = 0.19367182
Iteration 4, loss = 0.13252615
Iteration 5, loss = 0.10121732
Iteration 6, loss = 0.07471255
Iteration 7, loss = 0.05676647
Iteration 8, loss = 0.04415037
Iteration 9, loss = 0.03039997
Iteration 10, loss = 0.02703080
Iteration 1, loss = 1.30074268

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.44940235
Iteration 3, loss = 0.96491406
Iteration 4, loss = 0.44081133
Iteration 5, loss = 0.35481794
Iteration 6, loss = 0.53032771
Iteration 7, loss = 0.36314835
Iteration 8, loss = 0.31416079
Iteration 9, loss = 0.28943256
Iteration 10, loss = 0.26544893
Iteration 1, loss = 1.04867622

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 2, loss = 0.33709079
Iteration 3, loss = 0.26050449
Iteration 4, loss = 0.19984495
Iteration 5, loss = 0.17144745
Iteration 6, loss = 0.13618263
Iteration 7, loss = 0.65774883
Iteration 8, loss = 0.21093101
Iteration 9, loss = 0.16151059
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.83035768
Iteration 2, loss = 0.25720301
Iteration 3, loss = 0.82076672
Iteration 4, loss = 0.23261915
Iteration 5, loss = 0.15946379
Iteration 6, loss = 0.12189619
Iteration 7, loss = 0.18415185
Iteration 8, loss = 0.09925235
Iteration 9, loss = 0.11875773
Iteration 10, loss = 0.06922819

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

Iteration 1, loss = 0.72206052
Iteration 2, loss = 0.24458878
Iteration 3, loss = 0.40284047
Iteration 4, loss = 0.78453473
Iteration 5, loss = 0.19621946
Iteration 6, loss = 0.13955970
Iteration 7, loss = 0.11129606
Iteration 8, loss = 0.20055476
Iteration 9, loss = 0.51074099
Iteration 10, loss = 0.14146996
Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
Iteration 1, loss = 0.79218847
Iteration 2, loss = 0.68402770
Iteration 3, loss = 0.22962385
Iteration 4, loss = 0.16341120
Iteration 5, loss = 0.10921594
Iteration 6, loss = 0.07820200
Iteration 7, loss = 0.05614769
Iteration 8, loss = 0.04211918
Iteration 9, loss = 0.02918220
Iteration 10, loss = 0.02197686

D:\Anaconda3\lib\site-packages\sklearn\neural_network\multilayer_perceptron.py:563: ConvergenceWarning: Stochastic Optimizer: Maximum iterations reached and the optimization hasn't converged yet.
  % (), ConvergenceWarning)

train_score,test_score
Out[10]:
(array([[ 0.94864384,  0.93065126,  0.96410313,  0.90910101,  0.936     ,
          0.2704144 ,  0.34373958,  0.91146412,  0.72306154,  0.93002333],
        [ 0.98354824,  0.9717715 ,  0.97843965,  0.98355373,  0.97833333,
          0.4591712 ,  0.97144762,  0.9508998 ,  0.90557654,  0.9545707 ],
        [ 0.99777679,  0.9956657 ,  0.99788842,  0.99733304,  0.99544444,
          0.98144651,  0.9818909 ,  0.9329038 ,  0.90968674,  0.98611574],
        [ 0.99922188,  0.99888864,  0.99855523,  0.99866652,  0.99933333,
          0.93878458,  0.79346739,  0.99500111,  0.9958898 ,  0.95445962],
        [ 0.99833259,  0.99944432,  0.99955546,  0.99944438,  0.99955556,
          0.10087768,  0.53582935,  0.95267718,  0.99700067,  0.99822281]]),
 array([[ 0.89442231,  0.90419162,  0.87824351,  0.87412587,  0.913     ,
          0.26526527,  0.34834835,  0.87775551,  0.76753507,  0.91975928],
        [ 0.92131474,  0.94211577,  0.90818363,  0.93006993,  0.921     ,
          0.47847848,  0.93693694,  0.91082164,  0.91182365,  0.93681043],
        [ 0.94223108,  0.95508982,  0.91317365,  0.95004995,  0.948     ,
          0.92692693,  0.94194194,  0.91583166,  0.9258517 ,  0.95687061],
        [ 0.94621514,  0.96207585,  0.91317365,  0.94605395,  0.952     ,
          0.9019019 ,  0.78178178,  0.9498998 ,  0.97294589,  0.9217653 ],
        [ 0.9561753 ,  0.95908184,  0.92015968,  0.94805195,  0.956     ,
          0.1011011 ,  0.56456456,  0.9248497 ,  0.97294589,  0.96288867]]))
plt.plot(train_score)
Out[11]:
[<matplotlib.lines.Line2D at 0x91d3da0>,
 <matplotlib.lines.Line2D at 0x91d3f60>,
 <matplotlib.lines.Line2D at 0x91db198>,
 <matplotlib.lines.Line2D at 0x91db390>,
 <matplotlib.lines.Line2D at 0x91db588>,
 <matplotlib.lines.Line2D at 0x91db780>,
 <matplotlib.lines.Line2D at 0x91db978>,
 <matplotlib.lines.Line2D at 0x91dbb70>,
 <matplotlib.lines.Line2D at 0x91dbd68>,
 <matplotlib.lines.Line2D at 0x91dbf60>]
plt.plot(test_score)
Out[12]:
[<matplotlib.lines.Line2D at 0x929b710>,
 <matplotlib.lines.Line2D at 0x929b8d0>,
 <matplotlib.lines.Line2D at 0x929bac8>,
 <matplotlib.lines.Line2D at 0x929bcc0>,
 <matplotlib.lines.Line2D at 0x929beb8>,
 <matplotlib.lines.Line2D at 0x92a30f0>,
 <matplotlib.lines.Line2D at 0x92a32e8>,
 <matplotlib.lines.Line2D at 0x92a34e0>,
 <matplotlib.lines.Line2D at 0x92a36d8>,
 <matplotlib.lines.Line2D at 0x92a38d0>]