Search

Date Range
Advanced
Adjust the alpha value (0 to 1)
Adjust the relevance threshold (0 to 1)
www.kingswaysoft.com Tutorial

Preparing and Ingesting Databricks Lakehouse Data into Dataverse

The document outlines a data integration workflow from Databricks to Dataverse, focusing on data quality cleansing near the source. It uses a Databricks SQL View to standardize fields and validate records, with valid data loaded into Dataverse and invalid data routed to SQL Server for review, while preserving a centralized, reusable cleansing approach separate from downstream ingestion.

10 mins read Listed 29 days ago in Bluesky
vutr.substack.com Tutorial

ETL and ELT for data engineers - by Vu Trinh - VuTrinh.

The document explains ETL/ELT as data engineers' core task of moving data from multiple sources to a centralized data warehouse. It covers the data warehouse concept, its historical motivation, and the ETL protocol—extract, transform, then load—highlighting how data sources expanded beyond a single database and how transformation precedes loading.

2 mins read Listed 25 days ago in Substack: VuTrinh.
www.hpcwire.com Blog post

BigDATAwire - Data Science • AI • Advanced Analytics

Enterprises increasingly seek to reduce data-layer complexity by enabling direct querying across streaming, transactional, and analytical systems. Major vendors (Confluent, Google Cloud, EDB) announce features and acquisitions to unify data access (e.g., Kafka with Iceberg, PostgreSQL with ClickHouse) and improve data sovereignty, potentially reducing pipelines and data copies while raising concerns about vendor lock-in and partial readiness.

4 mins read Listed 8 days ago in Brave News
github.com Tutorial

local-agent-blueprint/docs/core/11-intent-driven-data-flows.md at main · MaxMSchneider/local-agent-blueprint · GitHub

Treat integrations as modular pipelines with separate adapters, normalization, intent specifications, processing, and sinks. Data shape and retention are defined by a human-owned intent spec, enabling multiple uses from the same data while keeping changes localized.

5 mins read Listed 14 days ago in reddit/LocalLLaMA/Please tell me Qwen3.8-Flash-Next can be made to be better than this
www.getdbt.com Blog post

How Kramp cut analyst onboarding from days to minutes with a self-serve dbt Mesh factory | dbt Labs

Kramp scaled dbt Mesh across its business using Data Project Factory to reduce onboarding and centralize governance, moving from a shadow BI setup to domain-led data products with self-serve infrastructure and federated controls. Over two years, they migrated 1,400 sources and 5,000 models, moved infrastructure from Dataform to dbt, and from AWS to GCP, achieving faster onboarding, improved time-to-insight, and higher developer satisfaction. They plan AI agents per domain tied to a unified semantic layer.

4 mins read Listed 19 days ago in RSS: dbt Labs Blog | Learn from the experts
github.com Tools and repositories

GitHub - bruin-data/bruin: Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows. · GitHub

Bruin is a data pipeline tool that unifies ingestion, multi-language transformations, and data quality within a single framework. It runs on local machines, EC2, or GitHub Actions, includes templating, isolation for Python, and end-to-end dry-run validation.

1 min read Listed 23 days ago in reddit/LLMDevs/Open-source tool to run LLM models on DWH data
r2dssanity.substack.com Documentation

From Task-Specific Data to Modeling Ready Data

The document explains that successful modeling starts with a task-specific framing of the business problem, then prepares a task-aligned dataset with a clear data dictionary. It describes the role of dd_parser_cleaner as the handoff that formalizes human decisions into a reusable data contract and metadata, enabling reliable downstream modeling; it also notes the tool may suggest types but cannot replace analyst judgment, and outlines the kinds of data transformations needed before training.

11 mins read Listed 26 days ago in reddit/learnmachinelearning/Moving from task ready data to modeling ready data
www.zoho.com Tools and repositories

Announcing Zoho DataPrep 3.0 | Build ETL pipelines faster with Agentic AI

Zoho DataPrep 3.0 adds Agentic Ask Zia for goal-based data pipelines, editable plans, and data enrichment with user approval. It also features a rebuilt engine, Python transforms, 10+ connectors, and the ability to run pipelines from external AI agents via MCP server.

5 mins read Listed 17 days ago in RSS: Zoho Blog
www.confluent.io Tutorial

A Tale of Two Houses: A Comparison Between Streamhouse and Lakehouse

The document compares Streamhouse and Lakehouse architectures, defining Streamhouse as a real-time, current-state data layer that propagates changes to consumers, while Lakehouse stores file-based data with a metadata layer for batch-processed analytics. It explains their distinct properties, failure modes, and how they can complement each other in a hybrid data architecture, including practical use cases and terminology like Lightning Tables and open Iceberg/Delta formats.

11 mins read Listed 10 days ago in RSS: Confluent: Data in motion
www.influxdata.com Tutorial

The FDAP Stack Explained | Learn Hub | InfluxData

FDAP is a stack of Apache components (Arrow, Flight, DataFusion, Parquet) enabling fast, columnar analytics. InfluxDB 3 uses FDAP; Arrow provides in-memory format, Flight moves data over the network, DataFusion handles SQL planning and execution, and Parquet serves as the interchange format.

8 mins read Listed 9 days ago in reddit/dataengineering/Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.