Skip to content

Commit dc3fee5

Browse files
committed
Update docs with expected behaviour in regards to skippable exceptions
Before this commit, the expected behaviour when a skippbale exception occurs in a fault tolerant chunk-oriented step was not documented in details. This commit update the docs and adds a sample for each case (when a skippable exception occurs during read, process and write). Resolves BATCH-2541
1 parent 2b4a87b commit dc3fee5

File tree

9 files changed

+475
-6
lines changed

9 files changed

+475
-6
lines changed

spring-batch-core/src/main/java/org/springframework/batch/core/step/builder/FaultTolerantStepBuilder.java

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright 2006-2018 the original author or authors.
2+
* Copyright 2006-2019 the original author or authors.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -344,7 +344,10 @@ public FaultTolerantStepBuilder<I, O> noSkip(Class<? extends Throwable> type) {
344344
}
345345

346346
/**
347-
* Explicitly request certain exceptions (and subclasses) to be skipped.
347+
* Explicitly request certain exceptions (and subclasses) to be skipped. These
348+
* exceptions (and their subclasses) might be thrown during any phase of the chunk
349+
* processing (read, process, write) but separate counts are made of skips on
350+
* read, process and write inside the step execution.
348351
*
349352
* @param type the exception type.
350353
* @return this for fluent chaining

spring-batch-core/src/main/java/org/springframework/batch/core/step/skip/SkipPolicy.java

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright 2006-2007 the original author or authors.
2+
* Copyright 2006-2019 the original author or authors.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -20,6 +20,7 @@
2020
*
2121
* @author Lucas Ward
2222
* @author Dave Syer
23+
* @author Mahmoud Ben Hassine
2324
*/
2425
public interface SkipPolicy {
2526

@@ -31,7 +32,7 @@ public interface SkipPolicy {
3132
* {@code skipCount&lt;0}. Implementations should avoid throwing any
3233
* undeclared exceptions.
3334
*
34-
* @param t exception encountered while reading
35+
* @param t exception encountered while processing
3536
* @param skipCount currently running count of skips
3637
* @return true if processing should continue, false otherwise.
3738
* @throws SkipLimitExceededException if a limit is breached

spring-batch-core/src/main/resources/org/springframework/batch/core/configuration/xml/spring-batch-3.0.xsd

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -841,6 +841,9 @@
841841
are declared as included take precedence over the same value if it is also excluded.
842842
Exceptions that are already marked as no-rollback
843843
are automatically skippable (but it doesn't hurt to add them again here).
844+
Exceptions (and their subclasses) that are declared might be thrown
845+
during any phase of the chunk processing (read, process, write) but separate counts
846+
are made of skips on read, process and write inside the step execution.
844847
]]>
845848
</xsd:documentation>
846849
</xsd:annotation>

spring-batch-docs/asciidoc/step.adoc

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -588,7 +588,9 @@ public Step step1() {
588588

589589
In the preceding example, a `FlatFileItemReader` is used. If, at any point, a
590590
`FlatFileParseException` is thrown, the item is skipped and counted against the total
591-
skip limit of 10. Separate counts are made of skips on read, process, and write inside
591+
skip limit of 10. Exceptions (and their subclasses) that are declared might be thrown
592+
during any phase of the chunk processing (read, process, write) but separate counts
593+
are made of skips on read, process, and write inside
592594
the step execution, but the limit applies across all skips. Once the skip limit is
593595
reached, the next exception found causes the step to fail. In other words, the eleventh
594596
skip triggers the exception, not the tenth.

spring-batch-samples/README.md

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ IO Sample Job | | | |
4444
[Restart Sample](#restart-sample) | | | X | | | | | | | |
4545
[Retry Sample](#retry-sample) | | X | | | | | | | | |
4646
[Skip Sample](#skip-sample) | X | | | | | | | | | |
47+
[Chunk Scanning Sample](#chunk-scanning-sample) | X | | | | | | | | | |
4748
[Trade Job](#trade-job) | | | | | | X | | | | |
4849

4950
The IO Sample Job has a number of special instances that show different IO features using the same job configuration but with different readers and writers:
@@ -812,6 +813,72 @@ The format for the transaction attribute specification is given in
812813
the Spring Core documentation (e.g. see the Javadocs for
813814
[TransactionAttributeEditor](https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/transaction/interceptor/TransactionAttributeEditor.html)).
814815

816+
### Chunk Scanning Sample
817+
818+
In a fault tolerant chunk-oriented step, when a skippable exception is thrown during
819+
item writing, the item writer (which receives a chunk of items) does not
820+
know which item caused the issue. Hence, it will "scan" the chunk item by item
821+
and only the faulty item will be skipped. Technically, the commit-interval will
822+
be re-set to 1 and each item will re-processed/re-written in its own transaction.
823+
824+
The `org.springframework.batch.sample.skip.SkippableExceptionDuringWriteSample` sample
825+
illustrates this behaviour:
826+
827+
* It reads numbers from 1 to 6 in chunks of 3 items, so two chunks are created: [1, 2 ,3] and [4, 5, 6]
828+
* It processes each item by printing it to the standard output and returning it as is.
829+
* It writes items to the standard output and throws an exception for item 5
830+
831+
The expected behaviour when an exception occurs at item 5 is that the second chunk [4, 5, 6] is
832+
scanned item by item. Transactions of items 4 and 6 will be successfully committed, while
833+
the one of item 5 will be rolled back. Here is the output of the sample with some useful comments:
834+
835+
```
836+
1. reading item = 1
837+
2. reading item = 2
838+
3. reading item = 3
839+
4. processing item = 1
840+
5. processing item = 2
841+
6. processing item = 3
842+
7. About to write chunk: [1, 2, 3]
843+
8. writing item = 1
844+
9. writing item = 2
845+
10. writing item = 3
846+
11. reading item = 4
847+
12. reading item = 5
848+
13. reading item = 6
849+
14. processing item = 4
850+
15. processing item = 5
851+
16. processing item = 6
852+
17. About to write chunk: [4, 5, 6]
853+
18. writing item = 4
854+
19. Throwing exception on item 5
855+
20. processing item = 4
856+
21. About to write chunk: [4]
857+
22. writing item = 4
858+
23. processing item = 5
859+
24. About to write chunk: [5]
860+
25. Throwing exception on item 5
861+
26. processing item = 6
862+
27. About to write chunk: [6]
863+
28. writing item = 6
864+
29. reading item = null
865+
```
866+
867+
* Lines 1-10: The first chunk is processed without any issue
868+
* Lines 11-17: The second chunk is read and processed correctly and is about to be written
869+
* Line 18: Item 4 is successfully written
870+
* Line 19: An exception is thrown when attempting to write item 5, the transaction is rolled back and chunk scanning is about to start
871+
* Lines 20-22: Item 4 is re-processed/re-written successfully in its own transaction
872+
* Lines 23-25: Item 5 is re-processed/re-written with an exception. Its transaction is rolled back and is skipped
873+
* Lines 26-28: Item 6 is re-processed/re-written successfully in its own transaction
874+
* Line 29: Attempting to read the next chunk, but the reader returns `null`:
875+
the datasource is exhausted and the step ends here
876+
877+
Similar examples show the expected behaviour when a skippable exception is thrown
878+
during reading and processing can be found in
879+
`org.springframework.batch.sample.skip.SkippableExceptionDuringReadSample`
880+
and `org.springframework.batch.sample.skip.SkippableExceptionDuringProcessSample`.
881+
815882
### Tasklet Job
816883

817884
The goal is to show the simplest use of the batch framework with a
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
/*
2+
* Copyright 2019 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package org.springframework.batch.sample.skip;
18+
19+
import java.util.Arrays;
20+
21+
import org.springframework.batch.core.Job;
22+
import org.springframework.batch.core.Step;
23+
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
24+
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
25+
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
26+
import org.springframework.batch.item.ItemProcessor;
27+
import org.springframework.batch.item.ItemReader;
28+
import org.springframework.batch.item.ItemWriter;
29+
import org.springframework.batch.item.support.ListItemReader;
30+
import org.springframework.context.annotation.Bean;
31+
import org.springframework.context.annotation.Configuration;
32+
33+
/**
34+
* @author Mahmoud Ben Hassine
35+
*/
36+
@Configuration
37+
@EnableBatchProcessing
38+
public class SkippableExceptionDuringProcessSample {
39+
40+
private final JobBuilderFactory jobBuilderFactory;
41+
42+
private final StepBuilderFactory stepBuilderFactory;
43+
44+
public SkippableExceptionDuringProcessSample(JobBuilderFactory jobBuilderFactory,
45+
StepBuilderFactory stepBuilderFactory) {
46+
this.jobBuilderFactory = jobBuilderFactory;
47+
this.stepBuilderFactory = stepBuilderFactory;
48+
}
49+
50+
@Bean
51+
public ItemReader<Integer> itemReader() {
52+
return new ListItemReader<Integer>(Arrays.asList(1, 2, 3, 4, 5, 6)) {
53+
@Override
54+
public Integer read() {
55+
Integer item = super.read();
56+
System.out.println("reading item = " + item);
57+
return item;
58+
}
59+
};
60+
}
61+
62+
@Bean
63+
public ItemProcessor<Integer, Integer> itemProcessor() {
64+
return item -> {
65+
if (item.equals(5)) {
66+
System.out.println("Throwing exception on item " + item);
67+
throw new IllegalArgumentException("Unable to process 5");
68+
}
69+
System.out.println("processing item = " + item);
70+
return item;
71+
};
72+
}
73+
74+
@Bean
75+
public ItemWriter<Integer> itemWriter() {
76+
return items -> {
77+
System.out.println("About to write chunk: " + items);
78+
for (Integer item : items) {
79+
System.out.println("writing item = " + item);
80+
}
81+
};
82+
}
83+
84+
@Bean
85+
public Step step() {
86+
return this.stepBuilderFactory.get("step")
87+
.<Integer, Integer>chunk(3)
88+
.reader(itemReader())
89+
.processor(itemProcessor())
90+
.writer(itemWriter())
91+
.faultTolerant()
92+
.skip(IllegalArgumentException.class)
93+
.skipLimit(3)
94+
.build();
95+
}
96+
97+
@Bean
98+
public Job job() {
99+
return this.jobBuilderFactory.get("job")
100+
.start(step())
101+
.build();
102+
}
103+
104+
}
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
/*
2+
* Copyright 2019 the original author or authors.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package org.springframework.batch.sample.skip;
18+
19+
import java.util.Arrays;
20+
21+
import org.springframework.batch.core.Job;
22+
import org.springframework.batch.core.Step;
23+
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
24+
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
25+
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
26+
import org.springframework.batch.item.ItemProcessor;
27+
import org.springframework.batch.item.ItemReader;
28+
import org.springframework.batch.item.ItemWriter;
29+
import org.springframework.batch.item.support.ListItemReader;
30+
import org.springframework.context.annotation.Bean;
31+
import org.springframework.context.annotation.Configuration;
32+
33+
/**
34+
* @author Mahmoud Ben Hassine
35+
*/
36+
@Configuration
37+
@EnableBatchProcessing
38+
public class SkippableExceptionDuringReadSample {
39+
40+
private final JobBuilderFactory jobBuilderFactory;
41+
42+
private final StepBuilderFactory stepBuilderFactory;
43+
44+
public SkippableExceptionDuringReadSample(JobBuilderFactory jobBuilderFactory,
45+
StepBuilderFactory stepBuilderFactory) {
46+
this.jobBuilderFactory = jobBuilderFactory;
47+
this.stepBuilderFactory = stepBuilderFactory;
48+
}
49+
50+
@Bean
51+
public ItemReader<Integer> itemReader() {
52+
return new ListItemReader<Integer>(Arrays.asList(1, 2, 3, 4, 5, 6)) {
53+
@Override
54+
public Integer read() {
55+
Integer item = super.read();
56+
System.out.println("reading item = " + item);
57+
if (item != null && item.equals(5)) {
58+
System.out.println("Throwing exception on item " + item);
59+
throw new IllegalArgumentException("Sorry, no 5 here!");
60+
}
61+
return item;
62+
}
63+
};
64+
}
65+
66+
@Bean
67+
public ItemProcessor<Integer, Integer> itemProcessor() {
68+
return item -> {
69+
System.out.println("processing item = " + item);
70+
return item;
71+
};
72+
}
73+
74+
@Bean
75+
public ItemWriter<Integer> itemWriter() {
76+
return items -> {
77+
System.out.println("About to write chunk: " + items);
78+
for (Integer item : items) {
79+
System.out.println("writing item = " + item);
80+
}
81+
};
82+
}
83+
84+
@Bean
85+
public Step step() {
86+
return this.stepBuilderFactory.get("step")
87+
.<Integer, Integer>chunk(3)
88+
.reader(itemReader())
89+
.processor(itemProcessor())
90+
.writer(itemWriter())
91+
.faultTolerant()
92+
.skip(IllegalArgumentException.class)
93+
.skipLimit(3)
94+
.build();
95+
}
96+
97+
@Bean
98+
public Job job() {
99+
return this.jobBuilderFactory.get("job")
100+
.start(step())
101+
.build();
102+
}
103+
104+
}

0 commit comments

Comments
 (0)