Troubleshooting Pending Files on Multiple Paths Due to Insufficient IOPS For TataPlay

Overview

The issue pertains to pending files on multiple paths on a server, as identified in a daily report. The root cause of the problem is insufficient Input/Output Operations Per Second (IOPS) of the /u01 partition. This leads to a delay in file processing and creates a backlog in the Parsing instance of the system.

Recording: Explaining the issue

Ticket: 4114625

Solution

To resolve the issue, follow these steps:

1. Create backlog directories to ensure a smooth flow of operations.
2. Check the IO Speed of the disk using the following commands:
   • For Read Speed: `dd if=/u01/testfile of=/dev/null bs=1M count=1000`
   • For Writing Speed: `dd if=/dev/zero of=/u01/testfile bs=1M count=1000`
3. Shift the application input path of the Processing service from **/u01/NATDATA/SYSLOG/JUNIPER/PRO_INPUT/ (SAS)** to **/opt/stlmediationsetup/Parsing_Root/input/ (SSD)**.
4. Move the backlog for reprocessing the files while maintaining the live flow.

After these changes, the backlog should be cleared and all services should be working correctly. Validate the daily report and the backlog count to confirm the resolution of the issue.

Summary

The issue of pending files on multiple paths due to insufficient IOPS of the /u01 partition can be resolved by creating backlog directories, checking the IO speed of the disk, shifting the application input path of the Processing service, and moving the backlog for reprocessing the files while maintaining the live flow.

FAQ

1. What is IOPS and why is it important?
   IOPS stands for Input/Output Operations Per Second. It is a common performance measurement used to benchmark computer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). Insufficient IOPS can lead to delays in file processing and create backlogs.
2. How can I check the IO speed of my disk?
   You can check the IO speed of your disk using the 'dd' command. For read speed, use `dd if=/u01/testfile of=/dev/null bs=1M count=1000` and for write speed, use `dd if=/dev/zero of=/u01/testfile bs=1M count=1000`.
3. What does shifting the application input path of the Processing service do?
   Shifting the application input path of the Processing service can help improve performance by moving the processing to a different location with better IOPS.