Failure Files observed for Distribution Service

Overview

The customer reports that the COUNTERSTATUS command in CMS for Distribution service shows "Failure Files" and the failure count is increasing sporadically

The same issue and solution applies if the customer reports that there is a mismatch or difference in count of files in Hourly Report and /u04/SFTP_backup:

[root@ODI-BHU-INFO-CGN-LGR-01 SFTP_backup]# ls -lrth | grep GP_20211211 | wc -l
46411

Summary ===> Total Files Received on 11/12/2021 = 47659 ===> Record Count = 8744239338 ===> Size = 113.624

Solution

Part 1: Identify the Issue

• Check the Logs for the Distribution service for which the issue is reported. The logs are located in $CRESTEL_P_ENGINE/module/mediation/logs/
  • Verify if there were any connectivity issues such as this. Such information is present in CrestelMediationServerLog.log:
  • 
• Check the Application Traces and check for connectivity issues such as this. Such information is usually present in CrestelMediationServerTraceLog.log:
  • 
• Run a ping to the SFTP Server and check latency issues/packet losses:
  • 
• If latency issues were seen or connectivity issues are seen, inform the customer to check network issues. 

Part 2: Process the failed files

• To process the failed files move them back from the failed directory to the input directory.
• The input and output directories are present in the Config directory of the service - Services > [service name] > [service ID] > [configuration file].xml
• Probably a cronjob can be implemented to do the copy in batches. Examples:
  • Example of command to move the files

    • find /data/distribution_root/error/input/FILE/2023/2/. -type f -iname "*.gz" -exec mv {} /data/distribution_root/input/ \;

• Example of cronjob to perform this in batches:
  • • 
    • The crontab entry for such a script:
    •