Skip to content

Commit 6c844bc

Browse files
csviricsviri
authored
docs: add page to describe cache access model (#2949)
1 parent 5dd1564 commit 6c844bc

File tree

3 files changed

+228
-0
lines changed

3 files changed

+228
-0
lines changed

‎docs/content/en/docs/documentation/_index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ This section contains detailed documentation for all Java Operator SDK features
1818
## Advanced Features
1919

2020
- **[Eventing](eventing/)** - Understanding the event-driven model
21+
- **[Accessing Resources in Caches](working-with-es-caches/) - How to access resources in caches
2122
- **[Observability](observability/)** - Monitoring and debugging your operators
2223
- **[Other Features](features/)** - Additional capabilities and integrations
2324

‎docs/content/en/docs/documentation/working-with-es-caches.md‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
---
2+
title: Working with EventSource caches
3+
weight: 48
4+
---
5+
6+
As described in [Event sources and related topics](eventing.md), event sources serve as the backbone
7+
for caching resources and triggering reconciliation for primary resources that are related
8+
to these secondary resources.
9+
10+
In the Kubernetes ecosystem, the component responsible for this is called an Informer. Without delving into
11+
the details (there are plenty of excellent resources online about informers), informers
12+
watch resources, cache them, and emit events when resources change.
13+
14+
`EventSource` is a generalized concept that extends the Informer pattern to non-Kubernetes resources,
15+
allowing you to cache external resources and trigger reconciliation when those resources change.
16+
17+
## The InformerEventSource
18+
19+
The underlying informer implementation comes from the Fabric8 client, called [DefaultSharedIndexInformer](https://github.com/fabric8io/kubernetes-client/blob/main/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/informers/impl/DefaultSharedIndexInformer.java).
20+
[InformerEventSource](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/InformerEventSource.java)
21+
in Java Operator SDK wraps the Fabric8 client informers.
22+
While this wrapper adds additional capabilities specifically required for controllers, this is the event
23+
source that most likely will be used to deal with Kubernetes resources.
24+
25+
These additional capabilities include:
26+
- Maintaining an index that maps secondary resources in the informer cache to their related primary resources
27+
- Setting up multiple informers for the same resource type when needed (for example, you need one informer per namespace if the informer is not watching the entire cluster)
28+
- Dynamically adding and removing watched namespaces
29+
- Other capabilities that are beyond the scope of this document
30+
31+
### Associating Secondary Resources to Primary Resource
32+
33+
Event sources need to trigger the appropriate reconciler, providing the correct primary resource, whenever one of their
34+
handled secondary resources changes. It is thus core to an event source's role to identify which primary resource
35+
(usually, your custom resource) is potentially impacted by that change.
36+
The framework uses [`SecondaryToPrimaryMapper`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java)
37+
for this purpose. For `InformerEventSources`, which target Kubernetes resources, this mapping is typically done using
38+
either the owner reference or an annotation on the secondary resource. For external resources, other mechanisms need to
39+
be used and there are also cases where the default mechanisms provided by the SDK do not work, even for Kubernetes
40+
resources.
41+
42+
However, once the event source has triggered a primary resource reconciliation, the associated reconciler needs to
43+
access the secondary resources which changes caused the reconciliation. Indeed, the information from the secondary
44+
resources might be needed during the reconciliation. For that purpose,
45+
`InformerEventSource` maintains a reverse
46+
index [PrimaryToSecondaryIndex](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/informer/DefaultPrimaryToSecondaryIndex.java),
47+
based on the result of the `SecondaryToPrimaryMapper`result.
48+
49+
## Unified API for Related Resources
50+
51+
To access all related resources for a primary resource, the framework provides an API to access the related
52+
secondary resources using the `Set<R> getSecondaryResources(Class<R> expectedType)` method of the `Context` object
53+
provided as part of the `reconcile` method.
54+
55+
For `InformerEventSource`, this will leverage the associated `PrimaryToSecondaryIndex`. Resources are then retrieved
56+
from the informer's cache. Note that since all those steps work on top of indexes, those operations are very fast,
57+
usually O(1).
58+
59+
While we've focused mostly on `InformerEventSource`, this concept can be extended to all `EventSources`, since
60+
[`EventSource`](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/EventSource.java#L93)
61+
actually implements the `Set<R> getSecondaryResources(P primary)` method that can be called from the `Context`.
62+
63+
As there can be multiple event sources for the same resource types, things are a little more complex: the union of each
64+
event source results is returned.
65+
66+
## Getting Resources Directly from Event Sources
67+
68+
Note that nothing prevents you from directly accessing resources in the cache without going through
69+
`getSecondaryResources(...)`:
70+
71+
```java
72+
public class WebPageReconciler implements Reconciler<WebPage> {
73+
74+
InformerEventSource<ConfigMap, WebPage> configMapEventSource;
75+
76+
@Override
77+
public UpdateControl<WebPage> reconcile(WebPage webPage, Context<WebPage> context) {
78+
// accessing resource directly from an event source
79+
var mySecondaryResource = configMapEventSource.get(new ResourceID("name","namespace"));
80+
// details omitted
81+
}
82+
83+
@Override
84+
public List<EventSource<?, WebPage>> prepareEventSources(EventSourceContext<WebPage> context) {
85+
configMapEventSource = new InformerEventSource<>(
86+
InformerEventSourceConfiguration.from(ConfigMap.class, WebPage.class)
87+
.withLabelSelector(SELECTOR)
88+
.build(),
89+
context);
90+
91+
return List.of(configMapEventSource);
92+
}
93+
}
94+
```
95+
96+
## The Use Case for PrimaryToSecondaryMapper
97+
98+
**TL;DR**: `PrimaryToSecondaryMapper` allows `InformerEventSource` to access secondary resources directly
99+
instead of using the `PrimaryToSecondaryIndex`. When this mapper is configured, `InformerEventSource.getSecondaryResources(..)`
100+
will call the mapper to retrieve the target secondary resources. This is typically required when the `SecondaryToPrimaryMapper`
101+
uses informer caches to list the target resources.
102+
103+
As discussed, we provide a unified API to access related resources using `Context.getSecondaryResources(...)`.
104+
The term "Secondary" refers to resources that a reconciler needs to consider when properly reconciling a primary
105+
resource. These resources encompass more than just "child" resources (resources created by a reconciler that
106+
typically have an owner reference pointing to the primary custom resource). They also include
107+
"related" resources (which may or may not be managed by Kubernetes) that serve as input for reconciliations.
108+
109+
In some cases, the SDK needs additional information beyond what's readily available, particularly when
110+
secondary resources lack owner references or any direct link to their associated primary resource.
111+
112+
Consider this example: a `Job` primary resource can be assigned to run on a cluster, represented by a
113+
`Cluster` resource.
114+
Multiple jobs can run on the same cluster, so multiple `Job` resources can reference the same `Cluster` resource. However,
115+
a `Cluster` resource shouldn't know about `Job` resources, as this information isn't part of what defines a cluster.
116+
When a cluster changes, though, we might want to redirect associated jobs to other clusters. Our reconciler
117+
therefore needs to determine which `Job` (primary) resources are associated with the changed `Cluster` (secondary)
118+
resource.
119+
See full
120+
sample [here](https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework/src/test/java/io/javaoperatorsdk/operator/baseapi/primarytosecondary).
121+
122+
```java
123+
InformerEventSourceConfiguration
124+
.from(Cluster.class, Job.class)
125+
.withSecondaryToPrimaryMapper(cluster ->
126+
context.getPrimaryCache()
127+
.list()
128+
.filter(job -> job.getSpec().getClusterName().equals(cluster.getMetadata().getName()))
129+
.map(ResourceID::fromResource)
130+
.collect(Collectors.toSet()))
131+
```
132+
133+
This configuration will trigger all related `Jobs` when the associated cluster changes and maintains the `PrimaryToSecondaryIndex`,
134+
allowing us to use `getSecondaryResources` in the `Job` reconciler to access the cluster.
135+
However, there's a potential issue: when a new `Job` is created, it doesn't automatically propagate
136+
to the `PrimaryToSecondaryIndex` in the `Cluster`'s `InformerEventSource`. Re-indexing only occurs
137+
when a `Cluster` event is received, which triggers all related `Jobs` again.
138+
Until this re-indexing happens, you cannot use `getSecondaryResources` for the new `Job`, since it
139+
won't be present in the reverse index.
140+
141+
You can work around this by accessing the Cluster directly from the cache in the reconciler:
142+
143+
```java
144+
145+
@Override
146+
public UpdateControl<Job> reconcile(Job resource, Context<Job> context) {
147+
148+
clusterInformer.get(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace()));
149+
150+
// omitted details
151+
}
152+
```
153+
154+
However, if you prefer to use the unified API (`context.getSecondaryResources()`), you need to add
155+
a `PrimaryToSecondaryMapper`:
156+
157+
```java
158+
clusterInformer.withPrimaryToSecondaryMapper( job ->
159+
Set.of(new ResourceID(job.getSpec().getClusterName(), job.getMetadata().getNamespace())));
160+
```
161+
162+
When using `PrimaryToSecondaryMapper`, the InformerEventSource bypasses the `PrimaryToSecondaryIndex`
163+
and instead calls the mapper to retrieve resources based on its results.
164+
In fact, when this mapper is configured, the `PrimaryToSecondaryIndex` isn't even initialized.
165+
166+
### Using Informer Indexes to Improve Performance
167+
168+
In the `SecondaryToPrimaryMapper` example above, we iterate through all resources in the cache:
169+
170+
```java
171+
context.getPrimaryCache().list().filter(job -> job.getSpec().getClusterName().equals(cluster.getMetadata().getName()))
172+
```
173+
174+
This approach can be inefficient when dealing with a large number of primary (`Job`) resources. To improve performance,
175+
you can create an index in the underlying Informer that indexes the target jobs for each cluster:
176+
177+
```java
178+
179+
@Override
180+
public List<EventSource<?, Job>> prepareEventSources(EventSourceContext<Job> context) {
181+
182+
context.getPrimaryCache()
183+
.addIndexer(JOB_CLUSTER_INDEX,
184+
(job -> List.of(indexKey(job.getSpec().getClusterName(), job.getMetadata().getNamespace()))));
185+
186+
// omitted details
187+
}
188+
```
189+
190+
where `indexKey` is a String that uniquely identifies a Cluster:
191+
192+
```java
193+
private String indexKey(String clusterName, String namespace) {
194+
return clusterName + "#" + namespace;
195+
}
196+
```
197+
198+
With this index in place, you can retrieve the target resources very efficiently:
199+
200+
```java
201+
202+
InformerEventSource<Job,Cluster> clusterInformer =
203+
new InformerEventSource(
204+
InformerEventSourceConfiguration.from(Cluster.class, Job.class)
205+
.withSecondaryToPrimaryMapper(
206+
cluster ->
207+
context
208+
.getPrimaryCache()
209+
.byIndex(
210+
JOB_CLUSTER_INDEX,
211+
indexKey(
212+
cluster.getMetadata().getName(),
213+
cluster.getMetadata().getNamespace()))
214+
.stream()
215+
.map(ResourceID::fromResource)
216+
.collect(Collectors.toSet()))
217+
.withNamespacesInheritedFromController().build(), context);
218+
```

‎operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/event/source/SecondaryToPrimaryMapper.java‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,16 @@
44

55
import io.javaoperatorsdk.operator.processing.event.ResourceID;
66

7+
/**
8+
* Maps secondary resource to primary resources.
9+
*
10+
* @param <R> secondary resource type
11+
*/
712
@FunctionalInterface
813
public interface SecondaryToPrimaryMapper<R> {
14+
/**
15+
* @param resource - secondary
16+
* @return set of primary resource IDs
17+
*/
918
Set<ResourceID> toPrimaryResourceIDs(R resource);
1019
}

0 commit comments

Comments
 (0)