Tech Trend Analysis System: Using Large Language Models and Finite State Chain Machines

Abstract

In today’s fast-paced technological environment, spotting emerging trends and anticipating future developments are important tasks in strategic planning and business decision-making. However, the volume and complexity of unstructured data containing relevant information make it very difficult for humans to effectively monitor, analyze, and identify inflection points by themselves. In this paper, we aim to prove the potential of integrating large language models (LLMs) with a novel finite state chain machine (FSCM) with output and graph databases to extract insights from unstructured data, specifically from earnings call transcripts of 40 top Technology Sector companies. The FSCM provides a modular, state-based approach for processing texts, enabling entity and relationship recognition. The extracted information is stored in a knowledge graph, further enabling semantic search and entity clustering. By leveraging this approach, we identified over 20,000 hidden (overlapping) trends and topics across various types. Our experiment on real-world datasets confirms the scalability and effectiveness of the method in extracting valuable knowledge from large datasets. The present work contributes to the field of Natural Language Processing (NLP) by showcasing the proposed method in addressing real-world business problems. The findings shed new light on current trends and challenges faced by tech companies, highlighting the potential for further integration with other NLP methods, leading to more robust and effective outcomes.