Skip to content

Commit 4974353

Browse files
snazyjbonofrerenovate-boteric-maynarddanielhumanmod
authored
Dremio merge 2025 05 12 15 00 (#64)
* Doc: Add release guide on the website (apache#1539) * main: Update actions/stale digest to f78de97 (apache#1547) * main: Update dependency boto3 to v1.38.12 (apache#1548) * main: Update postgres Docker tag to v17.5 (apache#1549) * main: Update dependency com.adobe.testing:s3mock-testcontainers to v4.2.0 (apache#1551) * main: Update dependency com.nimbusds:nimbus-jose-jwt to v10.3 (apache#1552) * Interface changes for pagination (apache#1528) * add missing apis * more tests, fixes * clean up drop * autolint * changes per review * revert iceberg messages to comply with oss tests * another revert * more iceberg catalog changes * autolint * dependency issues * more wiring * continuing rebase * remaining issues are related to task loading * re-add tests * debugging * fix failing tests * fix another test * changes per review * autolint * some fixes * stable * updates for new persistence * fix * continuing work * more reverts * continue reverts * more reverts * yank tests * autolint * test reverts * try to support limit without real page tokens * autolint * Stable * change comment * autolint * remove catalog config for now * changes per review * more tweaks * simplify types per review * Stable, about to refactor more * re-stable * polish * autolint * more changes per review * stable * Introduce reserved-properties setting; reserve "polaris." by default (apache#1417) * initial commit * initial commit * try to test * quarkus fixes * fix a bunch of callsites * Start applying changes * autolint * chase todos * autolint * bugfix * stable * add one test * stable with more tests * autolint * more tests * autolint * stable tests * clean up * oops * stabilize on main * autolint * more changes per review * Add cleanup support for partition-level statistics files when `DROP TABLE PURGE` (apache#1508) * cleaning up partition stats * update partition stat file extension * update test partition stat write impl * Implement federation to HadoopCatalog (apache#1466) * wip * quarkus fixes * autolint * hadoop impl * autolint * Refactors * refactored * autolint * add config * autolint * stable * Remove breakpoint anchor * add line to application.properties * yank HADOOP * autolint * Spark: Use builder for CreateGenericTableRequest instead of constructor for easier API spec update (apache#1546) * main: Update docker.io/jaegertracing/all-in-one Docker tag to v1.69.0 (apache#1559) * main: Update dependency io.opentelemetry:opentelemetry-bom to v1.50.0 (apache#1558) * main: Update dependency software.amazon.awssdk:bom to v2.31.40 (apache#1567) * main: Update dependency boto3 to v1.38.13 (apache#1556) * Include DISCLAIMER in binary distributions (apache#1568) * NoSQL: merge/rebase-fixes * NoSQL: bump dependencies * fix markdown --------- Co-authored-by: JB Onofré <[email protected]> Co-authored-by: Mend Renovate <[email protected]> Co-authored-by: Eric Maynard <[email protected]> Co-authored-by: danielhumanmod <[email protected]> Co-authored-by: gh-yzou <[email protected]>
1 parent e53ae00 commit 4974353

File tree

77 files changed

+2282
-352
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

77 files changed

+2282
-352
lines changed

‎.github/workflows/stale.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
stale:
2323
runs-on: ubuntu-24.04
2424
steps:
25-
- uses: actions/stale@816d9db1aba399a7f70277f1a2b01a4d21497fdd
25+
- uses: actions/stale@f78de9780efb7a789cf4745957fa3374cbb94fd5
2626
with:
2727
days-before-close: 5
2828
days-before-stale: 30

‎client/python/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ dependencies = [
3434
"python-dateutil>=2.8.2",
3535
"pydantic>=2.0.0",
3636
"typing-extensions>=4.7.1",
37-
"boto3==1.38.11",
37+
"boto3==1.38.13",
3838
]
3939

4040
[project.urls]

‎extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkMetaStoreSessionImpl.java‎

Lines changed: 29 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
import java.util.function.Predicate;
3838
import java.util.function.Supplier;
3939
import java.util.stream.Collectors;
40+
import java.util.stream.Stream;
4041
import org.apache.polaris.core.PolarisCallContext;
4142
import org.apache.polaris.core.context.RealmContext;
4243
import org.apache.polaris.core.entity.EntityNameLookupRecord;
@@ -52,6 +53,9 @@
5253
import org.apache.polaris.core.persistence.BaseMetaStoreManager;
5354
import org.apache.polaris.core.persistence.PrincipalSecretsGenerator;
5455
import org.apache.polaris.core.persistence.RetryOnConcurrencyException;
56+
import org.apache.polaris.core.persistence.pagination.HasPageSize;
57+
import org.apache.polaris.core.persistence.pagination.Page;
58+
import org.apache.polaris.core.persistence.pagination.PageToken;
5559
import org.apache.polaris.core.persistence.transactional.AbstractTransactionalPersistence;
5660
import org.apache.polaris.core.policy.PolarisPolicyMappingRecord;
5761
import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
@@ -419,29 +423,30 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
419423

420424
/** {@inheritDoc} */
421425
@Override
422-
public @Nonnull List<EntityNameLookupRecord> listEntitiesInCurrentTxn(
426+
public @Nonnull Page<EntityNameLookupRecord> listEntitiesInCurrentTxn(
423427
@Nonnull PolarisCallContext callCtx,
424428
long catalogId,
425429
long parentId,
426-
@Nonnull PolarisEntityType entityType) {
430+
@Nonnull PolarisEntityType entityType,
431+
@Nonnull PageToken pageToken) {
427432
return this.listEntitiesInCurrentTxn(
428-
callCtx, catalogId, parentId, entityType, Predicates.alwaysTrue());
433+
callCtx, catalogId, parentId, entityType, Predicates.alwaysTrue(), pageToken);
429434
}
430435

431436
@Override
432-
public @Nonnull List<EntityNameLookupRecord> listEntitiesInCurrentTxn(
437+
public @Nonnull Page<EntityNameLookupRecord> listEntitiesInCurrentTxn(
433438
@Nonnull PolarisCallContext callCtx,
434439
long catalogId,
435440
long parentId,
436441
@Nonnull PolarisEntityType entityType,
437-
@Nonnull Predicate<PolarisBaseEntity> entityFilter) {
442+
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
443+
@Nonnull PageToken pageToken) {
438444
// full range scan under the parent for that type
439445
return this.listEntitiesInCurrentTxn(
440446
callCtx,
441447
catalogId,
442448
parentId,
443449
entityType,
444-
Integer.MAX_VALUE,
445450
entityFilter,
446451
entity ->
447452
new EntityNameLookupRecord(
@@ -450,27 +455,33 @@ public List<EntityNameLookupRecord> lookupEntityActiveBatchInCurrentTxn(
450455
entity.getParentId(),
451456
entity.getName(),
452457
entity.getTypeCode(),
453-
entity.getSubTypeCode()));
458+
entity.getSubTypeCode()),
459+
pageToken);
454460
}
455461

456462
@Override
457-
public @Nonnull <T> List<T> listEntitiesInCurrentTxn(
463+
public @Nonnull <T> Page<T> listEntitiesInCurrentTxn(
458464
@Nonnull PolarisCallContext callCtx,
459465
long catalogId,
460466
long parentId,
461467
@Nonnull PolarisEntityType entityType,
462-
int limit,
463468
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
464-
@Nonnull Function<PolarisBaseEntity, T> transformer) {
469+
@Nonnull Function<PolarisBaseEntity, T> transformer,
470+
@Nonnull PageToken pageToken) {
465471
// full range scan under the parent for that type
466-
return this.store
467-
.lookupFullEntitiesActive(localSession.get(), catalogId, parentId, entityType)
468-
.stream()
469-
.map(ModelEntity::toEntity)
470-
.filter(entityFilter)
471-
.limit(limit)
472-
.map(transformer)
473-
.collect(Collectors.toList());
472+
Stream<PolarisBaseEntity> data =
473+
this.store
474+
.lookupFullEntitiesActive(
475+
localSession.get(), catalogId, parentId, entityType, pageToken)
476+
.stream()
477+
.map(ModelEntity::toEntity)
478+
.filter(entityFilter);
479+
480+
if (pageToken instanceof HasPageSize hasPageSize) {
481+
data = data.limit(hasPageSize.getPageSize());
482+
}
483+
484+
return Page.fromItems(data.map(transformer).collect(Collectors.toList()));
474485
}
475486

476487
/** {@inheritDoc} */

‎extension/persistence/eclipselink/src/main/java/org/apache/polaris/extension/persistence/impl/eclipselink/PolarisEclipseLinkStore.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
import org.apache.polaris.core.entity.PolarisEntityType;
3636
import org.apache.polaris.core.entity.PolarisGrantRecord;
3737
import org.apache.polaris.core.entity.PolarisPrincipalSecrets;
38+
import org.apache.polaris.core.persistence.pagination.PageToken;
3839
import org.apache.polaris.core.policy.PolarisPolicyMappingRecord;
3940
import org.apache.polaris.jpa.models.ModelEntity;
4041
import org.apache.polaris.jpa.models.ModelEntityActive;
@@ -282,7 +283,11 @@ long countActiveChildEntities(
282283
}
283284

284285
List<ModelEntity> lookupFullEntitiesActive(
285-
EntityManager session, long catalogId, long parentId, @Nonnull PolarisEntityType entityType) {
286+
EntityManager session,
287+
long catalogId,
288+
long parentId,
289+
@Nonnull PolarisEntityType entityType,
290+
@Nonnull PageToken pageToken) {
286291
diagnosticServices.check(session != null, "session_is_null");
287292
checkInitialized();
288293

‎extension/persistence/nosql/persistence/maintenance/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,5 +18,5 @@
1818
-->
1919

2020
Maintenance service,
21-
see [API package javadoc](api/src/main/java/org/apache/polaris/persistence/maintenance/api/package-info.java)
21+
see [API package javadoc](api/src/main/java/org/apache/polaris/persistence/nosql/maintenance/api/package-info.java)
2222
for information.

‎extension/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/PersistenceMetaStore.java‎

Lines changed: 32 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,8 @@
103103
import org.apache.polaris.core.persistence.dao.entity.LoadPolicyMappingsResult;
104104
import org.apache.polaris.core.persistence.dao.entity.PolicyAttachmentResult;
105105
import org.apache.polaris.core.persistence.dao.entity.PrincipalSecretsResult;
106+
import org.apache.polaris.core.persistence.pagination.Page;
107+
import org.apache.polaris.core.persistence.pagination.PageToken;
106108
import org.apache.polaris.core.policy.PolarisPolicyMappingRecord;
107109
import org.apache.polaris.core.policy.PolicyMappingUtil;
108110
import org.apache.polaris.core.policy.PolicyType;
@@ -1146,14 +1148,14 @@ Stream<ObjBase> listEntitiesStream(
11461148
long parentId,
11471149
PolarisEntityType entityType,
11481150
PolarisEntitySubType entitySubType,
1149-
long limit) {
1151+
PageToken pageToken) {
11501152

11511153
LOGGER.debug(
1152-
"listEntitiesStream, catalogId: {}, parentId: {}, entityType: {}, limit: {}",
1154+
"listEntitiesStream, catalogId: {}, parentId: {}, entityType: {}, pageToken: {}",
11531155
catalogStableId,
11541156
parentId,
11551157
entityType,
1156-
limit);
1158+
pageToken);
11571159

11581160
if (entityType == PolarisEntityType.NULL_TYPE) {
11591161
return Stream.empty();
@@ -1211,7 +1213,10 @@ Stream<ObjBase> listEntitiesStream(
12111213
objRefs = Streams.stream(nameIndex.iterator());
12121214
}
12131215

1214-
objRefs = objRefs.limit(limit);
1216+
// TODO PageToken ignored for now
1217+
// if (pageToken instanceof HasPageSize pageSize) {
1218+
// objRefs = objRefs.limit(pageSize.getPageSize());
1219+
// }
12151220

12161221
if (LOGGER.isDebugEnabled()) {
12171222
objRefs =
@@ -1244,68 +1249,67 @@ Stream<ObjBase> listEntitiesStream(
12441249
ListEntitiesResult listEntities(
12451250
List<PolarisEntityCore> catalogPath,
12461251
PolarisEntityType entityType,
1247-
PolarisEntitySubType entitySubType) {
1252+
PolarisEntitySubType entitySubType,
1253+
PageToken pageToken) {
12481254
var catalogStableId =
12491255
(catalogPath != null && !catalogPath.isEmpty()) ? catalogPath.getFirst().getId() : 0L;
12501256

12511257
var parentId =
12521258
(catalogPath != null && catalogPath.size() > 1) ? catalogPath.getLast().getId() : 0L;
12531259

12541260
var entities =
1255-
listEntitiesStream(catalogStableId, parentId, entityType, entitySubType, Long.MAX_VALUE)
1261+
listEntitiesStream(catalogStableId, parentId, entityType, entitySubType, pageToken)
12561262
.map(o -> mapToEntityNameLookupRecord(o, catalogStableId))
12571263
// TODO make PolarisTestMetaStoreManager handle non-mutable list :(
12581264
.collect(Collectors.toCollection(ArrayList::new));
12591265

1260-
return new ListEntitiesResult(entities);
1266+
return new ListEntitiesResult(entities, Optional.empty());
12611267
}
12621268

12631269
@Nonnull
12641270
@Override
1265-
public <T> List<T> listEntities(
1271+
public <T> Page<T> listEntities(
12661272
@Nonnull PolarisCallContext callCtx,
12671273
long catalogId,
12681274
long parentId,
12691275
@Nonnull PolarisEntityType entityType,
1270-
int limit,
12711276
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
1272-
@Nonnull Function<PolarisBaseEntity, T> transformer) {
1273-
return listEntitiesStream(
1274-
catalogId,
1275-
parentId,
1276-
entityType,
1277-
PolarisEntitySubType.ANY_SUBTYPE,
1278-
limit <= 0 ? Long.MAX_VALUE : limit)
1279-
.map(o -> mapToEntity(o, catalogId))
1280-
.filter(entityFilter)
1281-
.map(transformer)
1282-
.toList();
1277+
@Nonnull Function<PolarisBaseEntity, T> transformer,
1278+
@Nonnull PageToken pageToken) {
1279+
return Page.fromItems(
1280+
listEntitiesStream(
1281+
catalogId, parentId, entityType, PolarisEntitySubType.ANY_SUBTYPE, pageToken)
1282+
.map(o -> mapToEntity(o, catalogId))
1283+
.filter(entityFilter)
1284+
.map(transformer)
1285+
.toList());
12831286
}
12841287

12851288
@Nonnull
12861289
@Override
1287-
public List<EntityNameLookupRecord> listEntities(
1290+
public Page<EntityNameLookupRecord> listEntities(
12881291
@Nonnull PolarisCallContext callCtx,
12891292
long catalogId,
12901293
long parentId,
1291-
@Nonnull PolarisEntityType entityType) {
1292-
return listEntities(callCtx, catalogId, parentId, entityType, entity -> true);
1294+
@Nonnull PolarisEntityType entityType,
1295+
@Nonnull PageToken pageToken) {
1296+
return listEntities(callCtx, catalogId, parentId, entityType, entity -> true, pageToken);
12931297
}
12941298

12951299
@Nonnull
12961300
@Override
1297-
public List<EntityNameLookupRecord> listEntities(
1301+
public Page<EntityNameLookupRecord> listEntities(
12981302
@Nonnull PolarisCallContext callCtx,
12991303
long catalogId,
13001304
long parentId,
13011305
@Nonnull PolarisEntityType entityType,
1302-
@Nonnull Predicate<PolarisBaseEntity> entityFilter) {
1306+
@Nonnull Predicate<PolarisBaseEntity> entityFilter,
1307+
@Nonnull PageToken pageToken) {
13031308
return listEntities(
13041309
callCtx,
13051310
catalogId,
13061311
parentId,
13071312
entityType,
1308-
-1,
13091313
entityFilter,
13101314
entity ->
13111315
new EntityNameLookupRecord(
@@ -1314,7 +1318,8 @@ public List<EntityNameLookupRecord> listEntities(
13141318
entity.getParentId(),
13151319
entity.getName(),
13161320
entity.getTypeCode(),
1317-
entity.getSubTypeCode()));
1321+
entity.getSubTypeCode()),
1322+
pageToken);
13181323
}
13191324

13201325
@Override

‎extension/persistence/nosql/persistence/metastore/src/main/java/org/apache/polaris/persistence/nosql/metastore/PersistenceMetaStoreManager.java‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@
6262
import org.apache.polaris.core.persistence.dao.entity.PrivilegeResult;
6363
import org.apache.polaris.core.persistence.dao.entity.ResolvedEntityResult;
6464
import org.apache.polaris.core.persistence.dao.entity.ScopedCredentialsResult;
65+
import org.apache.polaris.core.persistence.pagination.PageToken;
6566
import org.apache.polaris.core.policy.PolicyEntity;
6667
import org.apache.polaris.core.policy.PolicyType;
6768
import org.apache.polaris.core.storage.PolarisStorageConfigurationInfo;
@@ -319,9 +320,10 @@ public ListEntitiesResult listEntities(
319320
@Nonnull PolarisCallContext callCtx,
320321
@Nullable List<PolarisEntityCore> catalogPath,
321322
@Nonnull PolarisEntityType entityType,
322-
@Nonnull PolarisEntitySubType entitySubType) {
323+
@Nonnull PolarisEntitySubType entitySubType,
324+
@Nonnull PageToken pageToken) {
323325
var ms = ms(callCtx);
324-
return ms.listEntities(catalogPath, entityType, entitySubType);
326+
return ms.listEntities(catalogPath, entityType, entitySubType, pageToken);
325327
}
326328

327329
@Nonnull
@@ -512,7 +514,7 @@ public EntityResult loadEntity(
512514
@Nonnull
513515
@Override
514516
public EntitiesResult loadTasks(
515-
@Nonnull PolarisCallContext callCtx, String executorId, int limit) {
517+
@Nonnull PolarisCallContext callCtx, String executorId, PageToken pageToken) {
516518
var ms = ms(callCtx);
517519

518520
// find all available tasks
@@ -522,7 +524,6 @@ public EntitiesResult loadTasks(
522524
PolarisEntityConstants.getRootEntityId(),
523525
PolarisEntityConstants.getRootEntityId(),
524526
PolarisEntityType.TASK,
525-
limit,
526527
entity -> {
527528
var taskState = PolarisObjectMapperUtil.parseTaskState(entity);
528529
long taskAgeTimeout =
@@ -536,11 +537,12 @@ public EntitiesResult loadTasks(
536537
|| taskState.executor == null
537538
|| callCtx.getClock().millis() - taskState.lastAttemptStartTime > taskAgeTimeout;
538539
},
539-
Function.identity());
540+
Function.identity(),
541+
PageToken.readEverything());
540542

541543
// TODO the following loop is NOT a "load" - it's a mutation over all loaded tasks !!
542544

543-
availableTasks.forEach(
545+
availableTasks.items.forEach(
544546
task -> {
545547
var originalTask = new PolarisBaseEntity(task);
546548
var properties =
@@ -558,7 +560,7 @@ public EntitiesResult loadTasks(
558560
task.setProperties(PolarisObjectMapperUtil.serializeProperties(callCtx, properties));
559561
ms.updateEntity(originalTask);
560562
});
561-
return new EntitiesResult(availableTasks);
563+
return new EntitiesResult(availableTasks.items);
562564
}
563565

564566
// Policies

0 commit comments

Comments
 (0)