Martin Körner

Data Engineering: Working with semi-structured data types. In our previous post, we demonstrated how to parse comma-separated data into a structured file format within the Snowflake project that utilizes the medallion architectural framework. In this post, we’ll introduce another file format that you’ll frequently encounter in your data engineering projects: the JSON File Format. JSON (JavaScript Object Notation) is an open standard file format used for sharing data. It employs human-readable text to store and transmit data objects, which consist of attribute-value pairs and arrays. This file format is commonly used for transmitting data between web applications and servers, making it very popular and suitable for our discussion and demonstration. To explain in more detail, we have three common types of JSON: JSON objects, JSON nested objects, and JSON arrays. JSON objects: “User_Name” :{ “Fist_Name”: “Henry”, “Last_Name”: “Godson” } Nested JSON objects: “User_Informations”: { “Date”: “2024-05-21”, “Sports”: {“Football”: “Good”, “Swimming”: “Excellent” } “Name”: “Henry } JSON arrays: (JSON are written inside the square brackets). “Employees”: [ { “First_Name”: “Henry”, “Last_Name”: “Godson” }, { “First_Name”: “Endy”, “Last_Name”: “Junior” }, ] We also have JSON Array in the simple format as “Favourit_Sports”: [“Football”, “Baseball”]. While certain data migration tools, such as Azure Data Factory, possess built-in intelligence for transforming data from one format to another, most data engineering ETL projects necessitate data engineers to perform similar transformations using either SQL or Python. To transform a JSON array using the Medallion Architectural Framework, we have two major approaches: Copy to Bronze and Transform: In this approach, we copy the JSON file into the Bronze layer and then transform the data to a structured format before moving it into the Silver table. Direct Transformation to Bronze: The second approach involves transforming the data directly to a structured format and depositing it into the Bronze layer straight from the staging environment. This approach is recommended, and we’ll demonstrate how to achieve it in our upcoming post. Stay tuned!

3

Tomorrow is the day, the Databricks Roundtable is taking place in Cologne.🚀 Join us for an exclusive hands-on session where Philipp Schwartenbeck will present a multi-agent system and Felix Xaver Krisch will demonstrate the complete lifecycle of a data product directly in Databricks! In the live demo, we will take you through the entire journey of a data project - starting with data exploration, followed by incremental data processing, workflow scheduling and scaling. We'll also show you how to implement CI/CD pipelines and monitoring tools to ensure seamless project delivery in Databricks. In the second live demo, we'll dive into the world of LLM-based multi-agent systems. See how autonomous collaboration between agents can solve complex tasks in real time, helping organisations improve the efficiency of their data processing and decision making! Find all the event details and register now... there are only 3 places left❗ 🔗https://lnkd.in/g_EyDuHp Looking forward to seeing you tomorrow! #databricks #dataengineering #AI #multiagent #LLM #cloudmigration #datagovernance #realtimedata #dataplatform #changemanagement

6

JSON Handling and Type Management with Large Datasets in Go -- Ep. 4 is out now! In this episode, Miki Tebeka tackles the complexities of working with JSON data, especially when dealing with extensive datasets and optimizing type management. Here, Miki introduces key strategies for managing large JSON documents, avoiding unnecessary type definitions, and using Go’s encoding/json package to handle custom serialization needs effectively. Things you will learn in this video: ✅ Streamlining JSON Handling: How to avoid excessive type declarations by using anonymous structures and selective decoding for large JSON datasets. ✅Implementing Custom Serialization: Techniques for defining custom JSON serialization in Go with the Marshaler and Unmarshaler interfaces. ✅Managing Mixed-Type Arrays: Strategies to work effectively with JSON arrays containing various data types, ensuring smooth integration with Go’s type system. 📰Watch Now: https://lnkd.in/ghjz58pC Stay tuned for more episodes from Miki Tebeka in this mini series, highlighting Intro to #JSON for Engineers & Managers.🤓 #golang

23

What if your data tools let you focus entirely on what matters — your use case? That’s exactly what Felix Krueger, senior data scientist at DZ Bank AG, experiences with Dataiku. ✨ The benefit: With no setup hassles or distractions, Felix simply logs into Dataiku, and everything he needs is ready. This simplicity allows him to focus on his use cases without extra technical overhead. 🔄 The transformation: Felix’s team at DZ Bank moved from isolated reporting solutions to a unified system within Dataiku. Now, they generate reports and dashboards across departments and even perform additional analysis, extending their previous workflows. ⭐ Favorite feature? The prepare recipe which, as he says, "just makes things so easy." 💡 Advice for companies starting with AI: Felix recommends having a use case to start but thinking ahead. “Start slow, but build an architecture that can scale over time” to prepare for future needs.

19

1 Comment

Next stop: Berlin, Germany 🇩🇪 "We've had failed data catalog implementations before. We realized all the potential falls apart when adoption struggles," shared a Berlin data leader, cutting straight to a universal truth in data governance. Their solution? Clear OKRs. "That's the only thing that has helped us align the entire organization toward creating the value we want," they explained. This focus on measurable outcomes has become their north star in driving adoption. But the conversation quickly turned to AI. "It's fascinating - just a few years ago, AI was just a buzzword. Now it's table stakes," they noted. The shift has been dramatic: users expect AI to unblock teams, eliminate manual effort, and deliver hyper-personalized solutions. The message from Berlin is clear: the race isn't just about implementing tools anymore - it's about meeting rapidly evolving user expectations while ensuring sustainable adoption across the organization.

20

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

Drowning in data silos? You're not alone. Fragmented data hinders access, analysis, and ultimately, innovation. But what if there was a better way? A Supergraph acts as a universal translator for your data, unifying disparate sources under a single, cohesive graph. Discover how APIwiz helps you to build and deploy efficient GraphQL APi Workflows. For more information, visit: www.apiwiz.io

14

The Nearest Neighbor Regressor is powerful, but scaling it efficiently is a challenge. Discover how KD Trees and Ball Trees enhance performance in high-dimensional spaces, making regression predictions smoother. Samy Baladram dives in! #MachineLearning #Regression

15

A LENS INTO MODERN DATA ENGINEERING "Building apps is like crafting furniture – each piece perfect. Data engineering, though, is like constructing an entire house." XUAN ANH NGUYEN (our data engineering lead) shares his journey from being our application developer to leading data engineering, rebuilding our project reporting system along the way. The Paradigm Shift: - From: Request/Response patterns - To: Data flows and transformations - Why: Scale, maintenance, and system thinking What We Learned: - Bottom-up vs. Top-down Design: Moving from feature-level focus to holistic system design. - Event Sourcing at Scale: Transitioning from CRUD to data streaming. - Processing Location Matters: Efficiency comes from processing near storage, not just speed. - Storage as a Backbone: Storage format choices, like Parquet, shape the entire pipeline’s efficiency and cost. Technical Validations: - Direct DuckDB queries outperformed traditional REST APIs - Parallel processing simplified with proper data partitioning - Simple S3 + Parquet + DuckDB pipeline handled our scale - Less infrastructure = more predictable systems Remember: In data engineering, the system is the feature. Your success isn’t measured by individual components working correctly, but by how well they work together to create value greater than the sum of their parts. #dwarves #dataengineering #systemdesign #architecture — Dwarves Notes (https://memo.d.foundation/) combine our team’s collective know-hows, R&Ds, and operation approaches. Connect and learn alongside other tech fellows: - Discord: discord.gg/dwarvesv - Github: https://lnkd.in/gZZ2eZMu - Website: https://d.foundation

4

Are you working with time series data and want to boost your forecasting game?📈 My latest blog post, "Unlocking Time Series Insights: Why Stationarity and Autocorrelation Matter?" dives into the fundamental concepts every data enthusiast needs to know: ✨ How stationarity simplifies analysis and sharpens your models. ✨ Why autocorrelation is the key to understanding patterns over time. ✨ What white noise tells us about randomness. Whether you’re a beginner or a pro, this post breaks down complex ideas into practical takeaways. Ready to unlock your time series potential? 👉 Read the full post here https://lnkd.in/e-Kgb-KQ

15

Tune into a new webinar with Tamara Janina Fingerlin to master Airflow and data engineering best practices! Learn to: 💡 Write smarter DAGs 💡 Master advanced scheduling 💡 Boost readability & maintenance Save your spot now! > https://shorturl.at/1IFpq Sponsor: Astronomer #Airflow #DataEngineering #Scheduling #Readability

2

Last week saw the General Availability of dynamic functionality for Databricks Workflows, in the form of the parameterized ForEach activity, but what does that mean? And why should we care? For a long time we've been using external orchestration tools whenever things had to be flexible, metadata driven or simply at a large scale - but with these changes we have a cheap, flexible way of achieving some pretty complex orchestration inside of Databricks itself! In this video Simon looks at a quick pattern for running a medallion-style lake processing routine, all driven by some JSON metadata! https://hubs.la/Q02NmWB_0 #DataEngineering #DataLakehouse #Spark #Azure #DataOrchestration #Finally

31

Build an effective recommender system seamlessly within your Snowflake environment using RelationalAI’s Snowflake Native App! Curious to see how it works? Check out this solution architecture on Snowflake Solution Center!

45

This week in our blog, we continue our series on Machine Learning Operations (MLOps) with a very interesting blog post about building an end-to-end MLOps pipeline with Databricks and GitHub Actions. Read it now! https://hubs.la/Q02zCl2J0 #data #analytics #machinelearning #mlops #databricks #github

15

4 Comments

Savant Spotlight Of The Week: Smart Stack 🗂️ Still wrangling with messy datasets? We know the feeling — trying to align columns and fix mismatches can be a real pain. What if merging data could be as easy as a couple of clicks? With #Savant Smart Stack, it is. Just upload your datasets, and let Smart Stack do the heavy lifting. It’ll sort out the mismatches, map your columns intelligently from various datasets and have everything ready in no time. 🕒 Save Time: No more endless hours merging and cleaning data. 💡 Intelligent Automation: Let our smart matching do the heavy lifting for you. 🗂️ Flexible Controls: Whether you want strict formats or quick-and-dirty merges, we’ve got you covered. No matter your need, combining datasets just became a two-click process. Quick and easy! 🚀 Try it? - https://buff.ly/3Z9lAr3 #SavantSpotlight #SavantTools #SmartStack #AnalyticsAutomation

20

Optimize your RAG pipeline with task-specific data ingestion & chunking techniques. Learn how to use metadata extraction, parent-document retrieval, & best practices for ensuring high-quality retrieval with our soon to be launched RAG ++ course: https://lnkd.in/ggWwT-ht

13