Home
Deep Learning For Vision

Apply RNN for Video Processing

Aim

To implement a simple Recurrent Neural Network (RNN) for video processing tasks such as action recognition or video classification by capturing temporal dependencies across video frames.

Algorithm

  1. Data Preparation:

    • Extract frames from videos.

    • Preprocess frames (resize, normalize, optionally extract features with CNN).

    • Form sequences of frames or features representing video clips.

  2. Model Architecture:

    • Use CNN (e.g., pretrained) to extract spatial features per frame.

    • Feed these sequential features into an RNN (e.g., LSTM or GRU) to model temporal relationships.

    • The RNN output passes through fully connected layers for classification.

  3. Training:

    • Train the network with labeled video sequences.

    • Use cross-entropy loss for classification and an optimizer like Adam.

  4. Evaluation:

    • Evaluate on validation or test video sequences.

  5. Inference:

    • Predict classes for unseen videos based on frame sequences.

Program (Python with TensorFlow/Keras)

python
import tensorflow as tf
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.layers import Input, LSTM, GRU, Dense, TimeDistributed, GlobalAveragePooling2D
import numpy as np

# Example: Simple video classification pipeline

# 1. CNN backbone for feature extraction (MobileNetV2)
cnn_base = MobileNetV2(weights='imagenet', include_top=False, input_shape=(64,64,3))
cnn_out = GlobalAveragePooling2D()(cnn_base.output)
cnn_model = Model(inputs=cnn_base.input, outputs=cnn_out)

# Freeze CNN layers (optional)
for layer in cnn_model.layers:
    layer.trainable = False

# 2. RNN model for temporal modeling
sequence_length = 10  # number of frames per video clip
feature_dim = 1280    # output feature size of MobileNetV2 GlobalAveragePooling2D
num_classes = 5       # example number of classes

model = Sequential([
    TimeDistributed(cnn_model, input_shape=(sequence_length, 64, 64, 3)),
    LSTM(64, return_sequences=False),
    Dense(32, activation='relu'),
    Dense(num_classes, activation='softmax')
])

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# 3. Example synthetic video data: 20 samples
X_train = np.random.rand(20, sequence_length, 64, 64, 3)
y_train = tf.keras.utils.to_categorical(np.random.randint(0, num_classes, 20), num_classes)

# 4. Train model (normally would use real video data)
model.fit(X_train, y_train, epochs=3, batch_size=2)

# 5. Predict on synthetic data
predictions = model.predict(X_train[:1])
print("Predicted class probabilities:", predictions)

Output

  • Training progress showing loss and accuracy over epochs.

  • Final printout of prediction probabilities for a sample video clip.

Epoch 1/3
10/10 [==============================] - 12s 780ms/step - loss: 1.5480 - accuracy: 0.2000
Epoch 2/3
10/10 [==============================] - 5s 529ms/step - loss: 1.4102 - accuracy: 0.4000
Epoch 3/3
10/10 [==============================] - 5s 536ms/step - loss: 1.2208 - accuracy: 0.5000
1/1 [==============================] - 0s 210ms/step
Predicted class probabilities: [[0.19557007 0.1330851  0.17529334 0.20284948 0.293202   ]]

Result

  • The model leverages CNN to extract spatial features per frame and RNN to learn temporal dynamics.

  • This approach captures motion and appearance patterns needed for video classification.

  • Although here shown on synthetic data, the method scales to real video tasks (e.g., action recognition, event detection).