-
Notifications
You must be signed in to change notification settings - Fork 143
Dlp #1909
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
russelmrcl
wants to merge
44
commits into
master
Choose a base branch
from
dlp
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Dlp #1909
Changes from all commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
bcc7dcb
add dlp
russelmrcl 497a43c
add comments
russelmrcl f0119d3
wip
russelmrcl 5f84eb6
refactor code
russelmrcl e2536fe
wip
russelmrcl 6ebb9c4
wip
russelmrcl 9d2e722
wip
russelmrcl 699b226
docs: minor
predic8 7ffee70
wip
russelmrcl 559a62b
wip
russelmrcl de3ed95
wip
russelmrcl 4c3ff8f
wip: dlp
russelmrcl 8cfb161
wip: dlp
russelmrcl b17b14a
wip: dlp
russelmrcl 099ec70
add test
russelmrcl 02d75ec
wip
russelmrcl 2be464b
add strategy pattern
russelmrcl 450204b
wip test
russelmrcl 75b56d9
wip
russelmrcl 8adedb9
wip
russelmrcl e7cc895
refactor code
russelmrcl 0f673bc
add tests
russelmrcl 6a0bed1
wip
russelmrcl 469256e
resolve conversations
russelmrcl 53bc2f8
convert to json parse
russelmrcl 7d04071
Merge branch 'master' into dlp
christiangoerdes ff15443
add path
russelmrcl ea1e179
wip
russelmrcl dce7806
add mask
russelmrcl e2eaa48
wip
russelmrcl 13711b1
wip
russelmrcl 7aa7e95
wip
russelmrcl 60a13df
fix
russelmrcl 0bf1268
refactor
russelmrcl 21c3489
wip
russelmrcl 20ed318
refactor code
russelmrcl c20c70d
improve log
russelmrcl 528cc14
add docs
russelmrcl 8d75f7c
edit docs
russelmrcl c3ec071
wip
russelmrcl e9a6286
refactor code
russelmrcl 8b1a278
refactor code
russelmrcl 77022dd
refactor code
russelmrcl 5008ee7
Merge branch 'master' into dlp
russelmrcl File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
20 changes: 20 additions & 0 deletions
20
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/Action.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import com.predic8.membrane.annot.MCAttribute; | ||
|
|
||
| public abstract class Action implements DLPAction { | ||
|
|
||
| private String field; | ||
|
|
||
| public String getField() { | ||
| return field; | ||
| } | ||
|
|
||
| @MCAttribute | ||
| public void setField(String field) { | ||
| this.field = field; | ||
| } | ||
|
|
||
| @Override | ||
| public abstract String apply(DLPContext context); | ||
| } | ||
78 changes: 78 additions & 0 deletions
78
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/CsvFieldConfiguration.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| import java.io.*; | ||
| import java.util.*; | ||
|
|
||
| public class CsvFieldConfiguration implements FieldConfiguration { | ||
|
|
||
| private static final Logger log = LoggerFactory.getLogger(CsvFieldConfiguration.class); | ||
|
|
||
| private final Map<String, String> riskLevels = new HashMap<>(); | ||
| private final Map<String, String> categories = new HashMap<>(); | ||
|
|
||
| @Override | ||
| public Map<String, String> getFields(String fileName) { | ||
| try (InputStream inputStream = getResourceAsStream(fileName)) { | ||
| BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream)); | ||
| String line; | ||
| boolean isHeader = true; | ||
|
|
||
| while ((line = reader.readLine()) != null) { | ||
| if (isHeader) { | ||
| isHeader = false; | ||
| continue; | ||
| } | ||
|
|
||
| line = line.trim(); | ||
| if (line.isEmpty() || line.startsWith("#")) continue; | ||
|
|
||
| String[] parts = line.split(",", -1); | ||
| if (parts.length < 3) { | ||
| log.warn("Skipping invalid line (less than 3 columns): {}", line); | ||
| continue; | ||
| } | ||
|
|
||
| String field = parts[0].trim().toLowerCase(Locale.ROOT); | ||
| String category = parts[1].trim(); | ||
| String riskLevel = parts[2].trim().toLowerCase(Locale.ROOT); | ||
|
|
||
| if (!isValidRiskLevel(riskLevel)) { | ||
| log.warn("Invalid risk level '{}' for field '{}'. Defaulting to 'unclassified'", riskLevel, field); | ||
| riskLevel = "unclassified"; | ||
| } | ||
|
|
||
| riskLevels.put(field, riskLevel); | ||
| categories.put(field, category); | ||
| } | ||
|
|
||
| } catch (IOException e) { | ||
| throw new RuntimeException("Error reading CSV field configuration: " + fileName, e); | ||
| } | ||
|
|
||
| return riskLevels; | ||
| } | ||
|
|
||
| public Map<String, String> getFieldCategories() { | ||
| return categories; | ||
| } | ||
|
|
||
| private InputStream getResourceAsStream(String fileName) { | ||
| InputStream is = CsvFieldConfiguration.class.getClassLoader().getResourceAsStream(fileName); | ||
| if (is == null) { | ||
| String msg = "Could not find CSV config file: " + fileName; | ||
| log.error(msg); | ||
| throw new IllegalArgumentException(msg); | ||
| } | ||
| return is; | ||
| } | ||
|
|
||
| private boolean isValidRiskLevel(String level) { | ||
| return switch (level) { | ||
| case "high", "medium", "low", "unclassified" -> true; | ||
| default -> false; | ||
| }; | ||
| } | ||
| } |
5 changes: 5 additions & 0 deletions
5
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/DLPAction.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| public interface DLPAction { | ||
| String apply(DLPContext context); | ||
| } |
90 changes: 90 additions & 0 deletions
90
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/DLPAnalyzer.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import com.fasterxml.jackson.core.JsonFactory; | ||
| import com.fasterxml.jackson.core.JsonFactoryBuilder; | ||
| import com.fasterxml.jackson.core.StreamReadConstraints; | ||
| import com.fasterxml.jackson.core.json.JsonReadFeature; | ||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.fasterxml.jackson.databind.ObjectMapper; | ||
| import com.predic8.membrane.core.http.Message; | ||
|
|
||
| import java.io.InputStream; | ||
| import java.util.*; | ||
|
|
||
| public class DLPAnalyzer { | ||
|
|
||
| private static final JsonFactory JSON_FACTORY = new JsonFactoryBuilder() | ||
| .configure(JsonReadFeature.ALLOW_TRAILING_COMMA, true) | ||
| .configure(JsonReadFeature.ALLOW_UNESCAPED_CONTROL_CHARS, false) | ||
| .streamReadConstraints(StreamReadConstraints.builder() | ||
| .maxNestingDepth(64) | ||
| .maxStringLength(16 * 1024) | ||
| .build()) | ||
| .build(); | ||
|
|
||
| private static final ObjectMapper MAPPER = new ObjectMapper(JSON_FACTORY); | ||
|
|
||
| private final Map<String, String> riskDict; | ||
| private final Map<String, String> categoryMap; | ||
|
|
||
| public DLPAnalyzer(Map<String, String> rawRiskMap, Map<String, String> categoryMap) { | ||
| this.riskDict = normalizeRiskLevels(rawRiskMap); | ||
| this.categoryMap = categoryMap; | ||
| } | ||
|
|
||
| private Map<String, String> normalizeRiskLevels(Map<String, String> raw) { | ||
| Map<String, String> result = new HashMap<>(); | ||
| raw.forEach((key, value) -> result.put(key, normalizeLevel(value))); | ||
| return result; | ||
| } | ||
|
|
||
| private String normalizeLevel(String level) { | ||
| switch (level.toLowerCase()) { | ||
| case "high": | ||
| case "medium": | ||
| case "low": | ||
| return level.toLowerCase(); | ||
| default: | ||
| return "unknown"; | ||
| } | ||
| } | ||
|
|
||
| public RiskReport analyze(Message msg) { | ||
| try (InputStream is = msg.getBodyAsStreamDecoded()) { | ||
| JsonNode root = MAPPER.readTree(is); | ||
| RiskReport report = new RiskReport(); | ||
| traverse(root, new ArrayDeque<>(), report); | ||
| return report; | ||
| } catch (Exception e) { | ||
| throw new RuntimeException("Failed to analyze message", e); | ||
| } | ||
| } | ||
|
|
||
| private void traverse(JsonNode node, Deque<String> path, RiskReport report) { | ||
| if (node.isObject()) { | ||
| node.fieldNames().forEachRemaining(fieldName -> { | ||
| path.addLast(fieldName); | ||
| traverse(node.get(fieldName), path, report); | ||
| path.removeLast(); | ||
| }); | ||
| } else if (node.isArray()) { | ||
| for (JsonNode child : node) { | ||
| traverse(child, path, report); | ||
| } | ||
| } else { | ||
| String fullPath = String.join(".", path); | ||
| String lastSegment = path.peekLast() != null ? path.peekLast() : ""; | ||
|
|
||
| String riskLevel = classify(fullPath, lastSegment); | ||
| String category = categoryMap.getOrDefault(fullPath, categoryMap.getOrDefault(lastSegment, "Unknown")); | ||
|
|
||
| report.recordField(fullPath, riskLevel, category); | ||
| } | ||
| } | ||
|
|
||
| private String classify(String fullPath, String simpleName) { | ||
| return Optional.ofNullable(riskDict.get(fullPath)) | ||
| .or(() -> Optional.ofNullable(riskDict.get(simpleName))) | ||
| .orElse("unknown"); | ||
| } | ||
| } |
8 changes: 8 additions & 0 deletions
8
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/DLPContext.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| public record DLPContext(String body, RiskReport riskReport) { | ||
|
|
||
| public boolean hasRiskReport() { | ||
| return riskReport != null; | ||
| } | ||
| } |
132 changes: 132 additions & 0 deletions
132
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/DLPInterceptor.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import com.predic8.membrane.annot.MCAttribute; | ||
| import com.predic8.membrane.annot.MCChildElement; | ||
| import com.predic8.membrane.annot.MCElement; | ||
| import com.predic8.membrane.core.exchange.Exchange; | ||
| import com.predic8.membrane.core.http.Message; | ||
| import com.predic8.membrane.core.interceptor.AbstractInterceptor; | ||
| import com.predic8.membrane.core.interceptor.Outcome; | ||
| import org.slf4j.Logger; | ||
| import org.slf4j.LoggerFactory; | ||
|
|
||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
|
|
||
| import static com.predic8.membrane.core.interceptor.Outcome.CONTINUE; | ||
|
|
||
| /** | ||
| * @description <p>Interceptor for Data Loss Prevention (DLP) in JSON-based request and response bodies.</p> | ||
| * | ||
| * <p>This plugin supports three main actions on sensitive fields:</p> | ||
| * <ul> | ||
| * <li> Mask ? Masks sensitive fields, leaving a configurable number of trailing characters visible.</li> | ||
| * <li> Filter? Removes specified fields entirely from the payload.</li> | ||
| * <li> Report ? Logs the risk level and category of specified fields (if configured).</li> | ||
| * </ul> | ||
| * | ||
| * @topic 3. Security and Validation | ||
| */ | ||
| @MCElement(name = "dlp") | ||
| public class DLPInterceptor extends AbstractInterceptor { | ||
|
|
||
| private static final Logger log = LoggerFactory.getLogger(DLPInterceptor.class); | ||
|
|
||
| private DLPAnalyzer dlpAnalyzer; | ||
| private String fieldsConfig; | ||
|
|
||
| private List<Mask> masks = new ArrayList<>(); | ||
| private List<Filter> filters = new ArrayList<>(); | ||
| private List<Report> reports = new ArrayList<>(); | ||
|
|
||
| private final List<DLPAction> actions = new ArrayList<>(); | ||
|
|
||
| @Override | ||
| public void init() { | ||
| if (fieldsConfig != null) { | ||
| CsvFieldConfiguration csv = new CsvFieldConfiguration(); | ||
| Map<String, String> levels = csv.getFields(fieldsConfig); | ||
| Map<String, String> cats = csv.getFieldCategories(); | ||
| this.dlpAnalyzer = new DLPAnalyzer(levels, cats); | ||
| } else { | ||
| this.dlpAnalyzer = new DLPAnalyzer(Map.of(), Map.of()); | ||
| } | ||
| actions.addAll(masks); | ||
| actions.addAll(filters); | ||
| actions.addAll(reports); | ||
| super.init(); | ||
| } | ||
|
|
||
| @Override | ||
| public Outcome handleRequest(Exchange exc) { | ||
| return handleInternal(exc.getRequest()); | ||
| } | ||
|
|
||
| private Outcome handleInternal(Message msg) { | ||
| try { | ||
| if (actions.isEmpty()) { | ||
| log.info("No actions configured. Skipping."); | ||
| return CONTINUE; | ||
| } | ||
| String body = msg.getBodyAsStringDecoded(); | ||
| RiskReport report = dlpAnalyzer.analyze(msg); | ||
| log.info("{}", report.getFormattedSummaryLog()); | ||
|
|
||
| for (DLPAction action : actions) { | ||
| body = action.apply(new DLPContext(body, report)); | ||
| } | ||
|
|
||
| msg.setBodyContent(body.getBytes(StandardCharsets.UTF_8)); | ||
| return CONTINUE; | ||
|
|
||
| } catch (Exception e) { | ||
| log.error("{}", e); | ||
| return Outcome.ABORT; | ||
| } | ||
| } | ||
|
|
||
| public String getFieldsConfig() { | ||
| return fieldsConfig; | ||
| } | ||
|
|
||
| /** | ||
| * @description Optionally, fields can be classified based on a CSV configuration. This file maps JSON paths or field names to risk levels and categories. | ||
| * @example fieldsConfig="dlp-fields.csv" | ||
| */ | ||
| @MCAttribute | ||
| public void setFieldsConfig(String fieldsConfig) { | ||
| this.fieldsConfig = fieldsConfig; | ||
| } | ||
|
|
||
| public List<Mask> getMasks() { | ||
| return masks; | ||
| } | ||
|
|
||
| @MCChildElement | ||
| public DLPInterceptor setMasks(List<Mask> masks) { | ||
| this.masks = masks; | ||
| return this; | ||
| } | ||
|
|
||
| public List<Filter> getFilters() { | ||
| return filters; | ||
| } | ||
|
|
||
| @MCChildElement(order = 1) | ||
| public DLPInterceptor setFilters(List<Filter> filters) { | ||
| this.filters = filters; | ||
| return this; | ||
| } | ||
|
|
||
| public List<Report> getReports() { | ||
| return reports; | ||
| } | ||
|
|
||
| @MCChildElement(order = 2) | ||
| public DLPInterceptor setReports(List<Report> reports) { | ||
| this.reports = reports; | ||
| return this; | ||
| } | ||
| } |
19 changes: 19 additions & 0 deletions
19
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/Field.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import com.predic8.membrane.annot.MCAttribute; | ||
| import com.predic8.membrane.annot.MCElement; | ||
|
|
||
| @MCElement(name = "field") | ||
| public class Field { | ||
|
|
||
| private String jsonpath; | ||
|
|
||
| public String getJsonpath() { | ||
| return jsonpath; | ||
| } | ||
|
|
||
| @MCAttribute | ||
| public void setJsonpath(String jsonpath) { | ||
| this.jsonpath = jsonpath; | ||
| } | ||
| } |
7 changes: 7 additions & 0 deletions
7
core/src/main/java/com/predic8/membrane/core/interceptor/dlp/FieldConfiguration.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| package com.predic8.membrane.core.interceptor.dlp; | ||
|
|
||
| import java.util.Map; | ||
|
|
||
| public interface FieldConfiguration { | ||
| Map<String, String> getFields(String fileName); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
Add validation for the field parameter.
The setter should validate that the field is not empty and is a valid JSONPath expression to prevent runtime errors.
@MCAttribute public void setField(String field) { + if (field != null && field.trim().isEmpty()) { + throw new IllegalArgumentException("field cannot be empty"); + } this.field = field; }🤖 Prompt for AI Agents