Part 4/10:
- Consumption Layer: Processes user inquiries, generates insights, and presents outputs in user-friendly formats.
All components are orchestrated via a custom-built automation orchestrator, ensuring smooth operation.
Processing Legacy SQL Scripts
Chunking: Large scripts (>2MB or high token count) are split into smaller, integrity-preserving chunks using scalable regular expressions.
Encryption & Privacy: All data is encrypted both at rest and during processing within Azure Data Lake.
Anonymization: Azure Language services anonymize hardcoded PII and sensitive data, preserving data privacy.
Metadata Extraction and Enrichment
- The system extracts procedural and functional metadata, including view definitions, triggers, and dynamic variables.