Optimize JdbcBasePersistenceImpl.listEntities

[XN137]

Is your feature request related to a problem? Please describe.

in #2317 a TODO got added to the jdbc persistence implementation:

polaris/persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java

Lines 424 to 443 in a8ca16b

	@Nonnull
	@Override
	public Page<EntityNameLookupRecord> listEntities(
	@Nonnull PolarisCallContext callCtx,
	long catalogId,
	long parentId,
	@Nonnull PolarisEntityType entityType,
	@Nonnull PolarisEntitySubType entitySubType,
	@Nonnull PageToken pageToken) {
	// TODO: only fetch the properties required for creating an EntityNameLookupRecord
	return listEntities(
	callCtx,
	catalogId,
	parentId,
	entityType,
	entitySubType,
	entity -> true,
	EntityNameLookupRecord::new,
	pageToken);
	}

currently the method is simply forwarding to the non-optimized listEntities variant.

Describe the solution you'd like

the idea is that in the JDBC query we should only load the columns/properties needed for creating a EntityNameLookupRecord instance.
the more general listEntities variant can be used as a starting point and trimmed down to just the necessary parts.

also according to this review comment we should make sure that the query in the optimized case can be served efficiently from the database by only having to read a single index (that contains all necessary columns) without having to access the main table/row storage (see also this explanation).

this may require a new or updated schema for the jdbc backend.

Describe alternatives you've considered

No response

Additional context

No response

[binarycat0]

Hello. I have capacity, please assign it to me.

[snazy]

Sure, feel free to work on it

[binarycat0]

The main changes of the Java code have been accepted and merged into master, but changes regarding the addition of a new JDBC Metastore index have been moved to discussion on the DEV mailinglist.
Please join the discussion and share your Pros/Cons in favor of adding the new index - [DISCUSS] Add JDBC Metastore table index

[XN137]

the java changes are in and the ML discussion for adding an index didnt seem to get much traction.
admins are free to add the index if they notice slow/inefficient queries around listing tables/namespaces.
so i'll mark this ticket as closed.