Add integration tests for table store #239
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @jnmt, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a comprehensive suite of integration tests for the table store functionality. It establishes a new common-test module to centralize shared test utilities and configurations, and then leverages this setup to validate various SQL-like operations (SELECT, INSERT, UPDATE, metadata queries, history queries) against the table store. Additionally, it includes minor refinements to the Scan contract to improve handling of primary and index key conditions.
Highlights
- New
common-testModule: A dedicated Gradle module (common-test) has been introduced to centralize shared test utilities and configurations, promoting reusability across different components. - Extensive Table Store Integration Tests: A new
TableStoreEndToEndTestclass has been added, providing thorough integration tests for the table store's SQL-like capabilities, covering data manipulation (INSERT, UPDATE, SELECT), metadata queries, and historical data retrieval. - Refined
ScanContract Logic: TheScancontract ingeneric-contractshas been updated to improve the validation logic for primary and index key conditions, specifically handlingIS NULLandIS NOT NULLoperators more robustly. - Gradle Build Configuration Updates: The build configurations for
client,table-store, andsettings.gradlehave been updated to incorporate the newcommon-testmodule and to define a dedicatedintegrationTestsource set and task for the table store.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive suite of integration tests for the table store, which is a valuable addition for ensuring its stability and correctness. The refactoring to create a common-test module for shared test utilities is also a good move towards better code organization. While the overall changes are positive, I've identified a few areas for improvement. Specifically, I've commented on a fragile path construction in the test base class, a brittle method for generating PartiQL query strings that should be refactored for robustness, and some misleading test method names that could cause confusion. Addressing these points will enhance the quality and maintainability of the new test suite.
| private String toPartiQLJsonString(JsonNode node) { | ||
| String result = jacksonSerDe.serialize(node); | ||
|
|
||
| // Remove double quotation in the key | ||
| result = FIELD_KEY.matcher(result).replaceAll("$1:"); | ||
|
|
||
| // Convert string value to '...' | ||
| result = STRING_VALUE.matcher(result).replaceAll("'$1'"); | ||
|
|
||
| // Escape single quotation | ||
| Matcher m = SINGLE_QUOTE_ESCAPE.matcher(result); | ||
| StringBuffer sb = new StringBuffer(); | ||
| while (m.find()) { | ||
| String inner = m.group(1).replace("'", "''"); | ||
| m.appendReplacement(sb, "'" + inner + "'"); | ||
| } | ||
| m.appendTail(sb); | ||
|
|
||
| return sb.toString(); | ||
| } |
There was a problem hiding this comment.
The implementation of toPartiQLJsonString is very fragile because it uses regular expressions to manipulate a serialized JSON string. This approach can easily fail with more complex JSON structures, such as nested objects or strings containing special characters (e.g., ") that are escaped by the JSON serializer.
A more robust approach is to build the PartiQL string by recursively traversing the JsonNode object. This avoids the pitfalls of string manipulation and correctly handles all valid JSON structures. Here is a suggested refactoring that achieves this:
private String toPartiQLJsonString(JsonNode node) {
if (node.isObject()) {
return "{" + java.util.stream.StreamSupport.stream(java.util.Spliterators.spliteratorUnknownSize(node.fields(), java.util.Spliterator.ORDERED), false)
.map(e -> e.getKey() + ":" + toPartiQLJsonString(e.getValue()))
.collect(java.util.stream.Collectors.joining(",")) + "}";
} else if (node.isArray()) {
return "[" + java.util.stream.StreamSupport.stream(java.util.Spliterators.spliteratorUnknownSize(node.elements(), java.util.Spliterator.ORDERED), false)
.map(this::toPartiQLJsonString)
.collect(java.util.stream.Collectors.joining(",")) + "]";
} else if (node.isTextual()) {
return "'" + node.asText().replace("'", "''") + "'";
} else {
return node.toString();
}
}There was a problem hiding this comment.
That's true. We will introduce a new feature/grammar that can handle native JSON without this kind of conversion in #246. So, let's keep as is for now.
| StorageFactory factory = StorageFactory.create(props); | ||
| storage = factory.getStorage(); | ||
| storageAdmin = factory.getStorageAdmin(); | ||
| ledgerSchemaPath = Paths.get(System.getProperty("user.dir") + LEDGER_SCHEMA_PATH); |
There was a problem hiding this comment.
The way ledgerSchemaPath is constructed is brittle. It relies on user.dir pointing to a specific directory relative to the schema file, which might not always be the case depending on how the tests are executed (e.g., from an IDE with a different working directory). This can lead to FileNotFoundException and flaky tests.
A more robust approach would be to place the schema file in the test resources and load it using the class loader. This would make the test setup independent of the execution environment.
For example, if ledger-schema.json is in src/main/resources of the common-test module, you could load it like this:
URL schemaUrl = getClass().getResource("/ledger-schema.json");
if (schemaUrl == null) {
throw new IllegalStateException("Could not find ledger-schema.json");
}
ledgerSchemaPath = Paths.get(schemaUrl.toURI());There was a problem hiding this comment.
It might be better for some cases, but we only use it in the tests in this repo, and it's a symlink shared in the schema-loader directory. So, I prefer to leave it as is.
table-store/src/integration-test/java/com/scalar/dl/tablestore/TableStoreEndToEndTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive integration tests for the table store module, providing end-to-end testing capabilities for SQL statement execution and validation operations.
- Adds a new
TableStoreEndToEndTestclass with extensive test coverage for SELECT, INSERT, UPDATE, and metadata operations - Configures integration test infrastructure including Gradle build configuration and CI/CD pipeline setup
- Creates a base test class
LedgerEndToEndTestBasefor common ledger testing functionality
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| table-store/src/integration-test/java/com/scalar/dl/tablestore/TableStoreEndToEndTest.java | Comprehensive integration test suite covering SQL operations, joins, history queries, and validation |
| table-store/build.gradle | Gradle configuration for integration test source set and task definition |
| common-test/src/main/java/com/scalar/dl/ledger/LedgerEndToEndTestBase.java | Base class providing common ledger test infrastructure and server setup |
| common-test/scripts/ledger-schema.json | Symlink to ledger schema for test configuration |
| common-test/build.gradle | Additional dependencies for ledger testing support |
| .github/workflows/scalar.yml | CI workflow configuration for running table store integration tests |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
table-store/src/integration-test/java/com/scalar/dl/tablestore/TableStoreEndToEndTest.java
Outdated
Show resolved
Hide resolved
table-store/src/integration-test/java/com/scalar/dl/tablestore/TableStoreEndToEndTest.java
Show resolved
Hide resolved
Description
This PR adds integration tests for the table store.
Related issues and/or PRs
Depends on:
Changes made
Checklist
Additional notes (optional)
N/A
Release notes
Added integration tests for the table store.