How to make AI training faster

Please follow and rate my GitHub repository.https://github.com/xinyuwei-david/david-share.git

Factors Affecting AI Learning Time

In deep learning training, the training time calculation includes many factors such as the number of epochs, global batch size, micro-batch size, and the number of computing devices. Below is the basic formula that describes the relationship between these parameters (this is only a basic explanation formula, and it mainly describes the proportional and inverse relationship. In actual training, more factors may need to be considered):

Among them—

• An epoch is the number of times a model processes the entire training data set.
• Total number of samples is the total number of samples in the training dataset.
• The global batch size is the total number of data samples processed in each training iteration.
• Step time is the time required for each training iteration, which varies depending on hardware performance, model complexity, optimization algorithm, and other factors.
• Number of devices is the number of computing devices used for training, such as the number of GPUs.

Note that while this formula provides a basic framework, actual training time may be affected by several other factors, including I/O speed, network latency (in case of distributed training), and CPU-GPU communication speed. Frequency of hardware errors during GPU learning Therefore, this formula can only be used as a rough estimate, and actual training times may vary.

Detailed description

The training time of a deep learning model is determined by several factors, including but not limited to:

• Number of Epochs: An epoch means that the model has processed the entire training data set once. The more epochs, the more data the model has to process, and therefore the longer the training time.
• Global Batch Size: The global batch size is the total number of data samples processed in each training iteration. A larger global batch size means more data is processed in each iteration, which reduces the number of iterations required per epoch, which can reduce the total training time. However, if the global batch size is too large, memory overflow can occur.
• Microbatch size: Microbatch size refers to the number of data samples that each computing device processes in each training iteration. The larger the microbatch size, the more data each device processes per iteration, which can improve computational efficiency and shorten training time. However, if the microbatch size is too large, memory overflow may occur.
• Hardware performance: The performance of the computing devices used (e.g. CPU, GPU) also affects the training time. A more powerful device can perform calculations faster, which can reduce training time.
• Model Complexity: The complexity of the model (e.g. number of layers, number of parameters, etc.) also affects the training time. The more complex the model, the more computation is required, and thus the longer the training time.
• Optimization Algorithm: The optimization algorithm used (SGD, Adam, etc.) and hyperparameter settings such as learning rate also affect the training time.
• Parallel Strategies: Using parallel computing strategies such as data parallelism, model parallelism, etc. will also impact training time.

There are many factors that determine training time and they should be considered comprehensively based on the specific training task and environment.

So in this formula

batch_size = 10  # Batch size  
total_num = 1000  # Total number of training data  

When training on one batch of data and updating the gradient once (gradient accumulation steps = 1):

train_steps = total_num / batch_size = 1000 / 10 = 100  

That is, there are 100 steps per epoch and the gradient update steps are also 100.
If memory is insufficient to support a batch size of 10, gradient accumulation can be used to reduce the size of each micro-batch. Assume that the gradient accumulation step is set to 2.

gradient_accumulation_steps = 2  
micro_batch_size = batch_size / gradient_accumulation_steps = 10 / 2 = 5  

This means that for each gradient update, we accumulate data in 2 micro-batches, each with a micro-batch size of 5. This reduces the memory overhead, but still keeps the data size at 10 data points per gradient update.

result:

• The number of training steps per epoch (train_steps) remains 100 because the total amount of data and the number of steps per epoch do not change.
• The number of gradient update steps is kept to 100 because each gradient update accumulates data from two micro-batches.

When using gradient accumulation, each learning step deals with accumulating gradients from multiple micro-batches, which may slightly increase the computation time per step. Therefore, if you have enough memory, it is recommended to increase the batch size to reduce the number of gradient accumulations. If you are short on memory, gradient accumulation is an effective method.

The global batch size has a significant impact on the learning effectiveness of the model. In general, a larger global batch size provides more accurate gradient estimates, which helps the model converge. However, it also increases the memory pressure of each device. If memory resources are limited, using a large global batch size may not be feasible.

In such cases, gradient accumulation can be used. By training with smaller micro-batch sizes on each device, we reduce memory pressure while maintaining a large global batch size for accurate gradient estimation. This allows training large models on limited hardware resources without sacrificing global batch size.

In summary, gradient accumulation is a trade-off strategy to balance global batch size and learning effectiveness when memory resources are limited.

Now let’s look at the following two formulas:

Assuming there is no OOM (out of memory) and the GPU compute power is not fully utilized, a larger global batch size leads to a shorter overall training time.

Relationship between data parallelism and batch size