Troubleshooting Pending Files Due to Insufficient IOPS on Server For TataPlay

Overview

The issue pertains to pending files on multiple paths on a server, as indicated by daily reports. The root cause is identified as insufficient Input/Output Operations Per Second (IOPS) of the /u01 partition, causing a delay in file processing and creating a backlog in the Parsing instance.

Recording: Explaining the issue

Ticket: 4114624

Solution

To resolve this issue, follow these steps:

1. Create backlog directories to ensure a smooth running flow.
2. Validate the IO Speed of the disk using the following commands:
   • For Read Speed: `dd if=/u01/testfile of=/dev/null bs=1M count=1000`
   • For Writing Speed: `dd if=/dev/zero of=/u01/testfile bs=1M count=1000`
3. Shift the application input path of the Processing service from **/u01/NATDATA/SYSLOG/JUNIPER/PRO_INPUT/ (SAS)** to **/opt/stlmediationsetup/Parsing_Root/input/ (SSD)**.
4. 
5. Move the backlog for reprocessing the files while keeping the live flow intact.

After these changes, validate the daily report and the backlog count. The system should now be working fine with no backlog.

Summary

The issue of pending files on multiple paths on a server due to insufficient IOPS of the /u01 partition can be resolved by creating backlog directories, validating the IO speed of the disk, shifting the application input path of the Processing service, and moving the backlog for reprocessing files while keeping the live flow intact.

FAQ

1. What is IOPS and why is it important?
   IOPS, or Input/Output Operations Per Second, is a common performance measurement used to benchmark computer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). When the IOPS is insufficient, it can cause delays in file processing and create a backlog.
2. How can I check the IO speed of my disk?
   You can check the IO speed of your disk using the `dd` command. For read speed, use `dd if=/u01/testfile of=/dev/null bs=1M count=1000`. For write speed, use `dd if=/dev/zero of=/u01/testfile bs=1M count=1000`.
3. What does shifting the application input path of the Processing service do?
   Shifting the application input path of the Processing service can help improve file processing speed. In this case, the path was shifted from a SAS partition to an SSD partition, as SSDs have higher IOPS.