Dated: May-2022 Over the years, I have lived to see application that were built and operating on systems like VAX/VMS, Sun system evolve into Windows and Unix systems, then to finally move into Cloud. However, Mainframes still continues to be the workhorse for many Financial instutions and also other verticals. …

Dated: April-2022 In my various engagements, Microsoft-Access databases (MDB/ACCDB) files are a common occurrence, apart from CSV, EXCEL, and PDFs. MS-Access is one of that well-loved software, which has been long used in the industry across various teams. It starts as a goto application for a quick application with forms…

Dated: Apr-2022 In this article, I demonstrate processing WebArchive (WARC) format files, natively in Snowflake using Java UDTF. Common Crawl Dataset The CommonCrawl.org builds and maintains an open repository of web crawl data that can be accessed and analyzed by anyone. The data corpus contains raw web page data, extracted metadata, and text extractions. …

Dated : Feb-2022 Over the recent years, it has always amazed me in how current softwares like Thoughtspot and Amazon Quicksight understands a query like ‘Who ran the most expensive query ?’ and responds back with the records and intelligently summarize the content. Sure Alexa, Siri, Google etc.. personnel assistants…

Observations on querying, TeraByte sized external table, with Snowflake. Dated : Feb-2022 Recently a client had a scenario where they have multi-terabytes of data hosted in thier object storage (S3). On any given day they source and stage around upwards of 10TB/day in the S3. …

My top recommendations for beginning your journey with DV2 — With the ever-changing landscape of source systems, modeling requirements, and data acquisition and integration options, the Data Vault 2.0 model provides the necessary patterns to adapt to these shifts. However, even if you are familiar with Data Vault 2 (DV2), it is not as simple as just understanding Hubs vs…

A Data Engineer’s Guide to Unlocking Creative Solutions with the Capabilities of Data Vault 2.0 — Not all clients are created equal, and with this logic in mind, neither are the methods for data acquisition and ingestion. One of the many challenges of working in data engineering is continuing to come up with creative solutions to solve your clients’ data problems. For example, if you have…

A simple walkthrough of Snowflake’s new Java UDF feature — In many of my Snowflake engagements, clients typically look to side-load data from Excel files. These files are either maintained by their internal business users or sourced from external data providers. …

Previously, I wrote a blog post on the workflow of deploying Snowflake external functions in AWS. Developing a cloud formation script could aid in the deployment process. Snowflake has provided cloud formation templates in this repo: sfguide-external-functions-examples. This template offers a good starting point. …