Keras：ResourceExhaustedError（参见上面的回溯）：OOM在分配形状的张量时[26671,32,32,64]

Question

我正在使用Keras on tensorflow后端（Keras版本2.1）训练我的网络，我在互联网上尝试了很多东西，但没有找到任何解决方案。

My Training set and labels: 26721(each image have size (32, 32,1)) , (26721, 10) 
Validation set and labels:  6680(each image have size(32,32,1), (6680,10)

到目前为止，这是我的模型，我使用的是Python3。

def CNN(input_, num_classes):

model = Sequential()

model.add(Convolution2D(16, kernel_size=(7, 7),  border_mode='same',
                 input_shape=input_))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(1, 1) ,  border_mode='same' ))
model.add(Convolution2D(64, (3, 3),  padding ='same'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=(1,1),  border_mode='same' ))
model.add(Flatten())
model.add(Dense(96))
model.add(Activation('relu'))

model.add(Dense(num_classes))
model.add(Activation('softmax'))
return model

model = CNN(image_size, num_classes)

model.compile(loss=keras.losses.categorical_crossentropy,
          optimizer=keras.optimizers.SGD(lr=0.01),
          metrics=['accuracy'])

print(model.summary())
csv_logger = CSVLogger('training.log')
early_stop = EarlyStopping('val_acc', patience=200, verbose=1)
model_checkpoint = ModelCheckpoint(model_save_path,
                                    'val_acc', verbose=0,
                                    save_best_only=True)

model_callbacks = [early_stop, model_checkpoint, csv_logger]
# print "len(train_dataset) ", len(train_dataset)
print("int(len(train_dataset)/batch_size) ", int(len(train_dataset)/batch_size))
K.get_session().run(tf.global_variables_initializer())
 model.fit_generator(train,
              steps_per_epoch=np.ceil(len(train_dataset)/batch_size),
              epochs=num_epochs,
              verbose=1,
              validation_data=valid,
              validation_steps=batch_size,
              callbacks=model_callbacks)

型号摘要：

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 32, 32, 16)        800       
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 32, 16)        64        
_________________________________________________________________
activation_1 (Activation)    (None, 32, 32, 16)        0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 32, 32, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 32, 32, 64)        9280      
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 32, 64)        256       
_________________________________________________________________
activation_2 (Activation)    (None, 32, 32, 64)        0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 32, 32, 64)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 65536)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 96)                6291552   
_________________________________________________________________
activation_3 (Activation)    (None, 96)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                970       
_________________________________________________________________
activation_4 (Activation)    (None, 10)                0         
=================================================================
Total params: 6,302,922
Trainable params: 6,302,762
Non-trainable params: 160

我根据批量大小发送图像。这是我的生成器功能：

# Generate images according to batch size


def gen(dataset, labels, batch_size):

images = []
digits = []
i = 0
while True:
    images.append(dataset[i])
    digits.append(labels[i]) 
    i+=1
    if i == batch_size:
        yield (np.array(images), np.array(digits))
        images = []
        digits = []
    # Generate remaining images also
    if i == len(dataset):
        yield (np.array(images), np.array(digits))
        images, digits = [], []
        i = 0

   train = gen(train_data, train_labels, batch_size)
   valid = gen(valid_data, valid_lables, batch_size)

错误登录终端：

请检查此链接是否有完整错误：Terminal Output

任何人都可以帮助我，我在这里做错了什么？

提前致谢

Answer 1

您正在整个火车上训练您的网络，这个火车太大而无法容纳在内存中，并且对于您的gpu来说太大了。

机器学习的标准是创建小批量的数据并对其进行训练。批量大小通常是16,32,64或其他一些2的幂，但它可以是任何东西，您通常必须通过交叉验证找到正确的批量大小。

Answer 2

从日志中可以看出，在分配edge_1094_loss之前，内存已经满了。检查值Limit ad InUse。

这可能是因为旧模型消耗了内存。快速解决这个问题就是简单地杀死进程。这将释放旧模型消耗的所有内存，这些内存不会被垃圾收集。