You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/articles/ai-the-context-engineer.md
+8-8Lines changed: 8 additions & 8 deletions
Original file line number
Diff line number
Diff line change
@@ -6,32 +6,32 @@ tags: ["ai", "data"]
6
6
categories: ["programming"]
7
7
---
8
8
9
-
With the advancements of LLM-based tooling, the responsibilities (and maybe even the title!) of data engineers is changing.
9
+
With the advancements of LLM-based tooling, the responsibilities (and maybe even the title!) of data engineers are changing.
10
10
11
11
## Origin
12
12
13
13
Before we talk about the changes, let's reflect on the inception of data engineering.
14
14
15
-
The practice of data engineering, and the need for dedicated roles, was born out of complexity, and the need for someone to manage it.
16
-
With an explosion in data volumes, and an increase in tools to manage it, a domain-specific engineer was born with a specialized set of skills.
15
+
The practice of data engineering, and the need for dedicated roles, was born out of complexity and the need for someone to manage it.
16
+
With an explosion in data volumes and an increase in tools to manage it, a domain-specific engineer was born with a specialized set of skills.
17
17
Some of these engineers came from the world of database administration, others from software engineering, and some from analytics.
18
-
As the domain that they worked in evolved, so did their responsibilities and skill-sets.
18
+
As the domain that they worked in evolved, so did their responsibilities and skillsets.
19
19
20
20
## Questions
21
21
22
-
But, while the tools and challenges that these engineers solved changed, the underlying need for their work remained the same: to support the answering of questions.
22
+
While the tools and challenges that these engineers solved changed, the underlying need for their work remained the same: to support the answering of questions.
23
23
24
24
- "Why do I have no customers?"
25
25
- "What is the significance of these protein structures?"
26
26
- "What is the airspeed velocity of an unladen swallow?"
27
27
28
-
The wrangling of data, storage, processing, and everything in-between was done out of necessity so that you, or some stakeholder, could make sense of it.
28
+
The wrangling of data, storage, processing, and everything inbetween was done out of necessity so that you, or some stakeholder, could make sense of it.
29
29
30
30
## Context
31
31
32
32
We're reaching another inflection point where the underlying tools are evolving.
33
-
This time, we're seeing an emergence of systems that can make sense of large volumes of data, however the new complexity is that these systems rely on carefully managed metadata, or _context_.
34
-
This context can store how data is loaded, how it's transformed, and how it's queried.
33
+
This time, we're seeing an emergence of systems that can make sense of large volumes of data. However the new "complexity" is that these systems rely on carefully managed metadata, or _context_.
34
+
This context can store how data is loaded, transformed, and queried.
35
35
You no longer have to write the tool-specific code to interface with various APIs, nor the queries to make sense of it.
36
36
You just have to ensure that the information is present and curated.
0 commit comments