Backlog seen in SFTP Uploader Input Path

Overview

The customer says that there is a backlog of 500K files in the /u02/prcs/EDR/PRO/Userdata directory. And this directory is the input path for the SFTP Uploader service:

The same solution applies if customer reports that Old files are being transferred to the Central Processing Server by the SFTP Uploader Service.

Solution

• Check Current SFTP Logs using the below Command

  • tail -f SFTPClient.log

  • The SFTP client logs are located under $SFTPUPLOADER_HOME/log/SFTPClient.log or /opt/crestelsetup/sftpuploader/log/SFTPClient.log
• In cases the backlog is high it could make the Service hang and it can be confirmed that the SFTPClient.log is not printing any new log entries. This indicates that there is a process hang.
• To resolve that first stop the SFTP Client Process using:

  • ps -ef |grep SFTP

  • kill -9 <PID returned from above command>

• Create New Directory in Parallel to the "UserData" input directory & move All the Pending Files to Another Directory. Example:

  • mkdir UserData_bkp

  • mv *.gz ../UserData_bkp

• Start SFTP Service as crestel user and check if the SFTP Client Logs are printing new entries:
  

  •  sh SFTPFileUploadRunner.sh ; tail -f ../logs/SFTPClient.log

Verification

• Verify that the SFTPClient.log is printing new entries. This is an indication that files are being processed/transferred.
• Verify that the count of files in the input directory is reducing thereby confirming that the file transfer is ongoing.
• For processing old files ask the customer to do batch processing by copying files in batches of 5-10K.