Time discrepant shipments in manifest data

Abstract

Manifest data is a log of container shipments from foreign lading ports to U.S. unlading ports. We provide several time varying network-based representations of this data in order to extract its most “discrepant” port pairs and contents patterns. We treat this time varying network representation as a combinatorial set system and use its discrepancy and firing rate (Abello et al. (2010) Detecting Novel Discrepancies in Communications Networks, International Conference on Data Mining, ICDM 2010: 8-17, Sydney, Australia and Chazelle (2000) The Discrepancy Method: Randomness and Complexity, Cambridge University Press) as the main statistics to track the most “salient” network elements. The output of the entire process is a “fossil” sub-network that encodes those port pairs and contents that exhibit unusual time varying patterns. It is expected that substantial deviations from these patterns will be useful triggers for further content inspections. The applicability of the proposed techniques is not limited to manifest data.