Skip to content

Commit 08488c1

Browse files
Davies Liurxin
authored andcommitted
[SPARK-5469] restructure pyspark.sql into multiple files
All the DataTypes moved into pyspark.sql.types The changes can be tracked by `--find-copies-harder -M25` ``` davieslocalhost:~/work/spark/python$ git diff --find-copies-harder -M25 --numstat master.. 2 5 python/docs/pyspark.ml.rst 0 3 python/docs/pyspark.mllib.rst 10 2 python/docs/pyspark.sql.rst 1 1 python/pyspark/mllib/linalg.py 21 14 python/pyspark/{mllib => sql}/__init__.py 14 2108 python/pyspark/{sql.py => sql/context.py} 10 1772 python/pyspark/{sql.py => sql/dataframe.py} 7 6 python/pyspark/{sql_tests.py => sql/tests.py} 8 1465 python/pyspark/{sql.py => sql/types.py} 4 2 python/run-tests 1 1 sql/core/src/main/scala/org/apache/spark/sql/test/ExamplePointUDT.scala ``` Also `git blame -C -C python/pyspark/sql/context.py` to track the history. Author: Davies Liu <[email protected]> Closes apache#4479 from davies/sql and squashes the following commits: 1b5f0a5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into sql 2b2b983 [Davies Liu] restructure pyspark.sql
1 parent d302c48 commit 08488c1

File tree

12 files changed

+2962
-2756
lines changed

12 files changed

+2962
-2756
lines changed

python/docs/pyspark.ml.rst

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,8 @@
11
pyspark.ml package
22
=====================
33

4-
Submodules
5-
----------
6-
7-
pyspark.ml module
8-
-----------------
4+
Module Context
5+
--------------
96

107
.. automodule:: pyspark.ml
118
:members:

python/docs/pyspark.mllib.rst

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,6 @@
11
pyspark.mllib package
22
=====================
33

4-
Submodules
5-
----------
6-
74
pyspark.mllib.classification module
85
-----------------------------------
96

python/docs/pyspark.sql.rst

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,18 @@
11
pyspark.sql module
22
==================
33

4-
Module contents
5-
---------------
4+
Module Context
5+
--------------
66

77
.. automodule:: pyspark.sql
88
:members:
99
:undoc-members:
1010
:show-inheritance:
11+
12+
13+
pyspark.sql.types module
14+
------------------------
15+
.. automodule:: pyspark.sql.types
16+
:members:
17+
:undoc-members:
18+
:show-inheritance:

python/pyspark/mllib/linalg.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929

3030
import numpy as np
3131

32-
from pyspark.sql import UserDefinedType, StructField, StructType, ArrayType, DoubleType, \
32+
from pyspark.sql.types import UserDefinedType, StructField, StructType, ArrayType, DoubleType, \
3333
IntegerType, ByteType
3434

3535

0 commit comments

Comments
 (0)