Having trouble using customized generator in fit validation

I’m writing a customized generator DataGenerator for model.fit(). It works good if I don’t specify validation_data parameter of model.fit().

python model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

If I try to specify validation_data parameter like following

python model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, validation_data=validation_generator.data_generator(), validation_steps=validation_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

I will get the warning W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled.

Train for 375 steps, validate for 93 steps Epoch 1/200 375/375 [==============================] – 1s 3ms/step – loss: 1.3730 – accuracy: 0.6803 – val_loss: 0.8896 – val_accuracy: 0.8324 Epoch 2/200 372/375 [============================>.] – ETA: 0s – loss: 0.7859 – accuracy: 0.83382021-05-25 13:37:10.626497: W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled 375/375 [==============================] – 1s 2ms/step – loss: 0.7854 – accuracy: 0.8338 – val_loss: 0.6558 – val_accuracy: 0.8585 Epoch 3/200 373/375 [============================>.] – ETA: 0s – loss: 0.6376 – accuracy: 0.85292021-05-25 13:37:11.257041: W tensorflow/core/kernels/data/ge nerator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled 375/375 [==============================] – 1s 2ms/step – loss: 0.6371 – accuracy: 0.8530 – val_loss: 0.5558 – val_accuracy: 0.8727

Is there any wrong with my generator?

Full code is

“`python import numpy as np import tensorflow as tf from tensorflow import keras

Network and training parameters

EPOCHS = 200 BATCH_SIZE = 128 VERBOSE = 1 CLASSES_NUM = 10 # Number of outputs = number of digits VALIDATION_SPLIT=0.2 # How much TRAIN is reserved for VALIDATION

class DataGenerator: def init(self, x, y, classes_num, batch_size=32, shuffle=True): ‘Initialization’ self.x = x self.y = y

 self.batch_size = batch_size self.classes_num = classes_num self.shuffle = shuffle self.steps = int(np.floor(len(self.x) / self.batch_size)) def __iter__(self): indexes = np.arange(len(self.x)) if self.shuffle == True: np.random.shuffle(indexes) for start in range(0, len(self.x), self.batch_size): end = min(start + self.batch_size, len(self.x)) idxes = indexes[start:end] batch_x = [self.x[idx] for idx in idxes] batch_y = [self.y[idx] for idx in idxes] yield np.array(batch_x), tf.keras.utils.to_categorical(batch_y, num_classes=self.classes_num) def data_generator(self): while True: yield from self.__iter__() 

Loading MNIST dataset

mnist = keras.datasets.mnist (x_train, y_train), (x_test, y_test) = mnist.load_data()

You can verify that the split between train and test is 60,000, and 10,000 respectively.

assert x_train.shape == (60000, 28, 28) assert x_test.shape == (10000, 28, 28) assert y_train.shape == (60000,) assert y_test.shape == (10000,)

X_train is 60000 rows of 28×28 values –> reshaped in 60000 x 784

RESHAPED = 784

x_train = x_train.reshape(x_train.shape[0], RESHAPED) x_test = x_test.reshape(x_test.shape[0], RESHAPED) x_train = x_train.astype(‘float32’) x_test = x_test.astype(‘float32’)

Normalize inputs to be within in [0, 1].

x_train /= 255 x_test /= 255

total = len(x_train)//BATCH_SIZE pivot = int((1-VALIDATION_SPLIT)*len(y_train)) x_train, x_validation = x_train[:pivot], x_train[pivot:] y_train, y_validation = y_train[:pivot], y_train[pivot:]

training_generator = DataGenerator(x_train, y_train, CLASSES_NUM, BATCH_SIZE) validation_generator = DataGenerator(x_validation, y_validation, CLASSES_NUM, BATCH_SIZE)

Build the model

model = tf.keras.models.Sequential() model.add(keras.layers.Dense(CLASSES_NUM, input_shape=(RESHAPED,), name=’dense_layer’, activation=’softmax’))

Summary of the model

model.summary()

Compile the model

model.compile(optimizer=’SGD’, loss=’categorical_crossentropy’, metrics=[‘accuracy’])

model.fit(training_generator.data_generator(), steps_per_epoch=training_generator.steps, validation_data=validation_generator.data_generator(), validation_steps=validation_generator.steps, epochs=EPOCHS, verbose=VERBOSE)

y_test = tf.keras.utils.to_categorical(y_test, CLASSES_NUM)

Evaluate the model

test_loss, test_acc = model.evaluate(x_test, y_test) print(‘Test loss:’, test_loss) print(‘Test accuracy:’, test_acc)

Make prediction

predictions = model.predict(x_test) “`