Overview
A high-concurrency analytics service that performs deep engine analysis on chess games from chess.com and lichess.org. The platform leverages Stockfish (a powerful open-source chess engine) within a serverless architecture to calculate proprietary performance metrics. The primary goal was to orchestrate the simultaneous analysis of up to 100 games per request using AWS Lambda, overcoming the challenges of running heavy binaries in a stateless environment.
Architecture & Workflow
Ingestion Layer: An API Gateway endpoint triggers an entry Lambda that accepts parameters (platform, player, game count, and analysis type). This function fetches game data in PGN (Portable Game Notation) format, serializes the data, and distributes individual game tasks into an SQS queue. It also initializes the record in DynamoDB.
Orchestration Layer: An SQS queue acts as a buffer, managing the distribution of tasks to the worker functions. This ensures system reliability and allows for controlled concurrency management.
Analysis Layer (The Worker): A specialized Python Lambda function processes individual games.
Custom Lambda Layers: Includes a uniquely compiled Stockfish binary and standard libraries.
Evaluation: The function executes the engine to analyze move quality and efficiency.
Scalability: Processes up to 100 games in parallel, with each execution updating its specific record in the DynamoDB table upon completion.
Retrieval Layer: An API Gateway and Lambda set that reads the calculated metrics from DynamoDB, provides a summary of the analysis, and tracks the real-time status of the batch calculation.
Technical Implementation
- Runtime Environment: Built with Python 3.11. Utilizes Lambda Layers to manage heavy dependencies including
pandas,requests, and the Stockfish binary. - Infrastructure as Code (IaC): Deployed using the Serverless Framework to manage the lifecycle of API Gateway, SQS, and DynamoDB.
- CI/CD Pipeline: Automated via GitHub Actions to build and publish Lambda layers and deploy service updates upon code merges.
- Data Strategy: Implemented Single Table Design in DynamoDB for efficient querying of game-level details and batch summaries.
Technologies Used
- AWS Services: API Gateway, SQS, Lambda, DynamoDB.
- Languages/Tools: Python 3.11, Stockfish, Serverless Framework, GitHub Actions.
- Chess Standards: PGN (Portable Game Notation).
Role
Lead Architect & Backend Developer