[Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories #23108

kevinyu98 · 2018-11-21T19:48:51Z

What changes were proposed in this pull request?

Add these test cases for resolution of ORC table location reported by SPARK-25993

also add corresponding test cases for Parquet table.

Update the

sql-migration-guide-update

doc

How was this patch tested?

This is a new test case.

dongjoon-hyun · 2018-11-22T10:18:20Z

ok to test

dongjoon-hyun · 2018-11-22T10:20:08Z

docs/sql-migration-guide-upgrade.md


  - Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's ORC data source table and ORC vectorization would be applied. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.

+  - In version 2.3 and earlier, `spark.sql.hive.converMetastoreOrc` default is `false`, if you specify a directory in the `LOCATION` clause in the `CREATE EXTERNAL TABLE STORED AS ORC LOCATION` sql statement, Spark will use the Hive ORC reader to read the data into the table if the directory or sub-directory contains the matching data, if you specify the wild card(*), the Hive ORC reader will not be able to read the data, because it is treating the wild card as a directory. For example: ORC data is stored at `/tmp/orctab1/dir1/`, `create external table tab1(...) stored as orc location '/tmp/orctab1/'` will read the data into the table, `create external table tab2(...) stored as orc location '/tmp/orctab1/*' ` will not. Since Spark 2.4, `spark.sql.hive.convertMetaStoreOrc` default is `true`, Spark will use native ORC reader, it will read the data if you specify the wild card, but will not if you specify the parent directory. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.


Could you change but will not if you specify the parent directory more clearly with examples like the other sentence?

dongjoon-hyun · 2018-11-22T10:23:08Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala

    }
  }
+
+  test("SPARK-25993 Add test cases for resolution of ORC table location") {


HiveOrcSourceSuite.scala will be the better place. And, we had better have the following and cover both case behaviors; true and false.

Seq(true, false).foreach { convertMetastore => withSQLConf(HiveUtils.CONVERT_METASTORE_ORC.key -> s"$convertMetastore") {

ok, I will move the test case to there. Thanks.

SparkQA · 2018-11-22T12:58:28Z

Test build #99179 has finished for PR 23108 at commit e238764.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-25T08:05:01Z

Test build #99235 has finished for PR 23108 at commit e731746.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-25T19:58:10Z

Retest this please.

SparkQA · 2018-11-25T22:21:02Z

Test build #99243 has finished for PR 23108 at commit e731746.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kevinyu98 · 2018-11-26T02:51:37Z

I fixed a typo in the testcase, retest please. Thanks.

SparkQA · 2018-11-26T06:25:05Z

Test build #99251 has finished for PR 23108 at commit d6e582b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-11-27T18:46:55Z

docs/sql-migration-guide-upgrade.md


  - Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's ORC data source table and ORC vectorization would be applied. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.

+  - In version 2.3 and earlier, `spark.sql.hive.converMetastoreOrc` default is `false`, if you specify a directory in the `LOCATION` clause in the `CREATE EXTERNAL TABLE STORED AS ORC LOCATION` sql statement, Spark will use the Hive ORC reader to read the data into the table if the directory or sub-directory contains the matching data, if you specify the wild card(*), the Hive ORC reader will not be able to read the data, because it is treating the wild card as a directory. For example: ORC data is stored at `/tmp/orctab1/dir1/`, `create external table tab1(...) stored as orc location '/tmp/orctab1/'` will read the data into the table, `create external table tab2(...) stored as orc location '/tmp/orctab1/*' ` will not. Since Spark 2.4, `spark.sql.hive.convertMetaStoreOrc` default is `true`, Spark will use native ORC reader, if you specify the wild card, it will try to read the matching data from current directory and sub-directory, if you specify a directory which does not contains the matching data, native ORC reader will not be able to read, even the data is in the sub-directory. For example: ORC data is stored at `/tmp/orctab1/dir1/`, `create external table tab3(...) stored as orc location '/tmp/orctab1/'` will not read the data from sub-directory into the table.  To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.


spark.sql.hive.converMetastoreOrc -> spark.sql.hive.convertMetastoreOrc

I've read this again. In fact, this is not a new behavior for Spark users because Apache Spark uses Parquet as a default format since 2.0 and the default behavior of STORED AS PARQUET works like this.

In order to give the richer context to the users and to avoid irrelevant confusions, we had better merge this part into the above line (line 112). For example, I'd like to update line 112 like the following.

applied. In addition, this makes Spark's Hive table read behavior more consistent over different formats and with the behavior of spark.read.load. For example, for both ORC/Parquet Hive tables, LOCATION '/table/*' is required instead of LOCATION '/table/' to create an external table reading its direct sub-directories like '/table/dir'. To set false to spark.sql.hive.convertMetastoreOrc restores the previous behavior.

Thanks, I will make changes.

dongjoon-hyun · 2018-11-27T20:34:06Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala

+             |  c2 int,
+             |  c3 string)
+             |STORED AS orc
+             |LOCATION '$wildCardDir'""".stripMargin


@kevinyu98 . This works, but there is a side effect with this. I mean this creates additional directory who name is '*' literally. And, newly inserted data go into that directory.

@dongjoon-hyun Hello Dongjoon, yes, you are right. It will create a directory with the name is '*', and it is the same behavior prior spark 2.4. I was just following the examples from the jira. Do you have any suggestions here? Thanks.

I have two suggestions.

Is this PR aiming only one-level subdirectories? Could you check the behavior on one, two, three level subdirectories in Parquet Hive tables first?

Since the test case looks general for both Parquet/ORC, please add a test case for Parquet while you are here.

@dongjoon-hyun Thanks for the suggestions. I tried with three level subdirectores for Parquet/ORC. Here is the result:

sql("set spark.sql.hive.convertMetastoreOrc=true")

three level directories

ORC:

"/*" can read sub directory data, but not three level subdirectories

"/" can only read current directory

Parquet:

"/*" can read sub directory data, but not three level subdirectories

"/" can only read current directory

sql("set spark.sql.hive.convertMetastoreOrc=false")

ORC:

"/" can read three level subdirectories

"/*" can't read any data

parquet:

"/" can only read current directory

"/*" can read sub directory data, but not three level subdirectories.

With sql("set spark.sql.hive.convertMetastoreOrc=true"), the ORC and Parquet behavior is consistent.

I think this PR is aiming only one-level subdirectores.

Sure, I will add one more for Parquet.

Thank you for investigating. I agree with you for (1). For the test case, please add three-level subdirectories. That will help us to improve Spark later. You may file another JIRA issue for that as a new feature JIRA.

@dongjoon-hyun Sorry for the delay. My got some issues with my Intellij environment. Sure, I will add three level subdirectories for this PR. FYI, I also tried with convertMetastoreParquet for Parquet, the behavior is consistent.
sql("set spark.sql.hive.convertMetastoreParquet = true")

three level

Parquet:

-- "/" can only read current directory
-- "/*" can read sub directory data, but not three level subdirectories.

sql("set spark.sql.hive.convertMetastoreParquet = false")

-- "/" can only read current directory
-- "/*" can read sub directory data, but not three level subdirectories.

@dongjoon-hyun I forgot to add three level subdirectores in the last commit, I am adding now, will submit soon.

Thanks, @kevinyu98 . Also, please update the title and description of this PR and SPARK-25993 JIRA issue.

- [Spark-25993][SQL][TEST]Add test cases for resolution of ORC table location + [SPARK-25993][SQL][TEST] Add test cases for CREATE EXTERNAL TABLE with subdirectories

SparkQA · 2018-12-03T23:14:39Z

Test build #99622 has finished for PR 23108 at commit d75b923.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-12-05T04:23:53Z

Test build #99688 has finished for PR 23108 at commit fe472c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-12-05T06:27:06Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

    }
  }
+
+  test("SPARK-25993 Add test cases for resolution of Parquet table location") {


Maybe, move to HiveParquetSourceSuite? That's the similar one with HiveOrcSourceSuite.

Also, let's replace the test case name with CREATE EXTERNAL TABLE with subdirectories.

Also, for the full test coverage, can we have the following combination like ORC, too?

Seq(true, false).foreach { convertMetastore =>

sure, changed.

dongjoon-hyun · 2018-12-05T06:29:26Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

    }
  }
+
+  test("SPARK-25993 Add test cases for resolution of ORC table location") {


Please change this to CREATE EXTERNAL TABLE with subdirectories, too.

dongjoon-hyun · 2018-12-05T06:31:56Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+
+  test("SPARK-25993 Add test cases for resolution of Parquet table location") {
+    withTempPath { path =>
+        val someDF1 = Seq((1, 1, "parq1"), (2, 2, "parq2")).toDF("c1", "c2", "c3").repartition(1)


Indentation.

dongjoon-hyun · 2018-12-05T06:32:07Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+    withTempPath { path =>
+        val someDF1 = Seq((1, 1, "parq1"), (2, 2, "parq2")).toDF("c1", "c2", "c3").repartition(1)
+        withTable("tbl1", "tbl2", "tbl3") {
+        val dataDir = s"${path.getCanonicalPath}/l3/l2/l1/"


indentation?

dongjoon-hyun · 2018-12-05T06:32:23Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

+          (1 to 2).map(i => Row(i, i, s"parq$i")))
+
+        val wildcardL3Statement =
+            s"""


indentation?

dongjoon-hyun · 2018-12-05T06:34:35Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcSourceSuite.scala

    }
  }

+  protected def testORCTableLocation(isConvertMetastore: Boolean): Unit = {


Since this test helper function is only used in HiveOrcSourceSuite, can we move this into HiveOrcSourceSuite?

ok, I moved.

SparkQA · 2018-12-06T00:31:40Z

Test build #99746 has finished for PR 23108 at commit 51d1d78.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-12-06T08:26:23Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

          |OPTIONS (
-          |  PATH '${new File(orcTableAsDir.getAbsolutePath).toURI}'
+          |  PATH '${new File(orcTableAsDir.getAbsolutePath
+        ).toURI}'


The above change in line 76 ~ 80 looks strange and irrelevant. Let's revert this change.

good catch, I didn't notice my changes affect the formatting in the file. I have revert the change. Thanks

dongjoon-hyun · 2018-12-06T08:36:28Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala

+                  } catch {
+                    case e: IOException =>
+                      assert(e.getMessage().contains("java.io.IOException: Not a file"))
+                  }


@kevinyu98 . Is this testing exceptions for the above all three SQLs? We use intercept[IOException] to test expected Exceptions.

For now, this looks like not a robust test case, because there is no assert(false) after sql(sqlStmt). We need to check the individual query failure and success exactly for the specific configuration.

you are right, I will make changes. Thanks.

dongjoon-hyun · 2018-12-06T08:39:04Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala

+              checkAnswer(sql(wildcardL3SqlStatement), Nil)
+            }
+          } finally {
+            hiveClient.runSqlHive("DROP TABLE IF EXISTS hive_orc")


It seems that we need to clean up tbl1 ~ tbl4, too.

@dongjoon-hyun at line 221, I put the tbl1 ~ tbl4 with the withTable, I think it will get dropped. I tried to run it couple time in intellij, it seems work fine. what do you think?

Got it. I missed that.

dongjoon-hyun · 2018-12-06T08:40:31Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala

 class HiveParquetSourceSuite extends ParquetPartitioningTest {
  import testImplicits._
  import spark._
+  import java.io.IOException


This had better go to line 20.

SparkQA · 2018-12-07T02:21:48Z

Test build #99799 has finished for PR 23108 at commit d851169.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

kevinyu98 · 2018-12-07T03:03:01Z

retest please

dilipbiswal · 2018-12-07T04:47:15Z

retest this please

SparkQA · 2018-12-07T07:17:04Z

Test build #99804 has finished for PR 23108 at commit d851169.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

It seems that we are testing the only data in depth 3.

1) tbl1 LOCATION '${path}/l3/l2/'
2) tbl2 LOCATION '${path}/l3/l2/*'
3) tbl3 LOCATION '${path}/l3/*'

Like the followings, the test coverage should have depth 1 case, too. (as I requested before)

4) tbl4 LOCATION '${path}/*' 
5) tbl5 LOCATION '${path}'

Let's have multiple data directories. Current one only puts the data into a single directory. Multiple data directory will represent the real use cases.
Could you make these Parquet/ORC test cases a little bit more similarily? We had better have the same style for the same test coverage. Currently, they look different. The best way is having a helper function, but it seems that it's gone during previous commits.

dongjoon-hyun · 2018-12-09T02:53:27Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveParquetSourceSuite.scala

    }
  }
+
+  test("SPARK-25993 CREATE EXTERNAL TABLE with subdirectories") {


Please fix this first for the first and second review comments.

@dongjoon-hyun Thanks for the comments, I have tried to make the changes for the first and second review comments, I changed both suites to make it looks similar, also add more test cases. For the 3rd comments, I haven't found a common place to both suites, when you say the help function missing in the previous commit, can you help to point what kind of help function I missed? Thanks.

SparkQA · 2018-12-12T21:56:52Z

Test build #100042 has finished for PR 23108 at commit fef8c68.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2018-12-12T22:53:41Z

retest this please

SparkQA · 2018-12-13T01:16:01Z

Test build #100049 has finished for PR 23108 at commit fef8c68.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

kevinyu98 · 2018-12-13T03:10:48Z

Retest please. I looked at the Test build, saw Builder #10055 has failure, but when I run the failure test case local, it works.
SparkPullRequestBuilder #100055
build/sbt "test-only org.apache.spark.sql.hive.client.HiveClientSuites"
info] Run completed in 15 minutes, 21 seconds.
[info] Total number of tests run: 189
[info] Suites: completed 18, aborted 0
[info] Tests: succeeded 189, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 189, Failed 0, Errors 0, Passed 189

dilipbiswal · 2018-12-13T06:19:08Z

retest this please

SparkQA · 2018-12-13T08:05:02Z

Test build #100072 has finished for PR 23108 at commit fef8c68.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2018-12-13T18:54:07Z

retest this please

SparkQA · 2018-12-14T00:04:35Z

Test build #100111 has finished for PR 23108 at commit fef8c68.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2019-06-11T08:03:31Z

cc @dilipbiswal @kevinyu98

AmplabJenkins · 2019-09-16T18:17:30Z

Can one of the admins verify this patch?

github-actions · 2020-01-04T00:06:29Z

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just
a way of keeping the PR queue manageable.

If you'd like to revive this PR, please reopen it!

kevinyu98 · 2020-01-08T07:02:10Z

I did the rebase on the master, open a new pr27130. There is no sql-migration-guide-upgrade.md on the master code, it is on the branch 2.4, should I update doc on branch 2.4? thanks

…th subdirectories ### What changes were proposed in this pull request? This PR aims to add these test cases for resolution of ORC table location reported by [SPARK-25993](https://issues.apache.org/jira/browse/SPARK-25993) also add corresponding test cases for Parquet table. ### Why are the changes needed? The current behavior is complex, this test case suites are designed to prevent the accidental behavior change. This pr is rebased on master, the original pr is [23108](#23108) ### Does this PR introduce any user-facing change? No. This adds test cases only. ### How was this patch tested? This is a new test case. Closes #27130 from kevinyu98/spark-25993-2. Authored-by: Kevin Yu <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

kevinyu98 added 2 commits November 21, 2018 08:28

add test case for orc table location

4e45ef9

doc the change in migration-guide

e238764

dongjoon-hyun reviewed Nov 22, 2018

View reviewed changes

address comments

e731746

fix typo

d6e582b

srowen approved these changes Nov 26, 2018

View reviewed changes

dongjoon-hyun reviewed Nov 27, 2018

View reviewed changes

kevinyu98 added 2 commits November 28, 2018 14:15

address comment

39ebcf8

adding testcase for parquet

d75b923

kevinyu98 changed the title ~~[Spark-25993][SQL][TEST]Add test cases for resolution of ORC table location~~ [Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories Dec 5, 2018

change to three level testing

fe472c8

dongjoon-hyun reviewed Dec 5, 2018

View reviewed changes

move the test cases

51d1d78

dongjoon-hyun reviewed Dec 6, 2018

View reviewed changes

address comments

d851169

dongjoon-hyun reviewed Dec 9, 2018

View reviewed changes

address comments

fef8c68

github-actions bot added the Stale label Jan 4, 2020

github-actions bot closed this Jan 5, 2020

kevinyu98 mentioned this pull request Jan 8, 2020

[SPARK-25993][SQL][TESTS] Add test cases for CREATE EXTERNAL TABLE with subdirectories #27130

Closed


		- Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's ORC data source table and ORC vectorization would be applied. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.

		- In version 2.3 and earlier, `spark.sql.hive.converMetastoreOrc` default is `false`, if you specify a directory in the `LOCATION` clause in the `CREATE EXTERNAL TABLE STORED AS ORC LOCATION` sql statement, Spark will use the Hive ORC reader to read the data into the table if the directory or sub-directory contains the matching data, if you specify the wild card(), the Hive ORC reader will not be able to read the data, because it is treating the wild card as a directory. For example: ORC data is stored at `/tmp/orctab1/dir1/`, `create external table tab1(...) stored as orc location '/tmp/orctab1/'` will read the data into the table, `create external table tab2(...) stored as orc location '/tmp/orctab1/' ` will not. Since Spark 2.4, `spark.sql.hive.convertMetaStoreOrc` default is `true`, Spark will use native ORC reader, it will read the data if you specify the wild card, but will not if you specify the parent directory. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.

[Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories #23108

[Spark-25993][SQL][TEST]Add test cases for CREATE EXTERNAL TABLE with subdirectories #23108

Uh oh!

Conversation

kevinyu98 commented Nov 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Nov 22, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 22, 2018

Uh oh!

SparkQA commented Nov 25, 2018

Uh oh!

dongjoon-hyun commented Nov 25, 2018

Uh oh!

SparkQA commented Nov 25, 2018

Uh oh!

kevinyu98 commented Nov 26, 2018

Uh oh!

SparkQA commented Nov 26, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinyu98 Dec 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Dec 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 3, 2018

Uh oh!

SparkQA commented Dec 5, 2018

Uh oh!

dongjoon-hyun Dec 5, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

kevinyu98 commented Nov 21, 2018 •

edited

Loading

dongjoon-hyun Nov 27, 2018 •

edited

Loading

dongjoon-hyun Nov 27, 2018 •

edited

Loading

kevinyu98 Dec 3, 2018 •

edited

Loading

dongjoon-hyun Dec 4, 2018 •

edited

Loading

dongjoon-hyun Dec 5, 2018 •

edited

Loading

dongjoon-hyun Dec 6, 2018 •

edited

Loading