From eb777a62be0001a7857a9a4a768c816695516ed9 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Mon, 7 May 2018 16:07:18 +0200 Subject: [PATCH 01/34] Create CODE_OF_CONDUCT.md --- CODE_OF_CONDUCT.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 CODE_OF_CONDUCT.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..df5363d --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,46 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at ben.hermann@uni-paderborn.de. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] + +[homepage]: http://contributor-covenant.org +[version]: http://contributor-covenant.org/version/1/4/ From 930a97548aabe68b1881213bc38b4b56e578fae1 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Mon, 7 May 2018 16:29:21 +0200 Subject: [PATCH 02/34] Issue templates --- .github/ISSUE_TEMPLATE/Bug_report.md | 38 +++++++++++++++++++++++ .github/ISSUE_TEMPLATE/Feature_request.md | 18 +++++++++++ 2 files changed, 56 insertions(+) create mode 100644 .github/ISSUE_TEMPLATE/Bug_report.md create mode 100644 .github/ISSUE_TEMPLATE/Feature_request.md diff --git a/.github/ISSUE_TEMPLATE/Bug_report.md b/.github/ISSUE_TEMPLATE/Bug_report.md new file mode 100644 index 0000000..d201fc1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/Bug_report.md @@ -0,0 +1,38 @@ +--- +name: Bug report +about: Create a report to help us improve + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Affected components (if known)** +Choose those applicable here: crawler, cli, webapi, webapp, management + +**Desktop (please complete the following information, if relevant):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] + +**Smartphone (please complete the following information, if relevant):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/Feature_request.md b/.github/ISSUE_TEMPLATE/Feature_request.md new file mode 100644 index 0000000..835eff2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/Feature_request.md @@ -0,0 +1,18 @@ +--- +name: Feature request +about: Suggest an idea for this project + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. +Just using "none" is okay here. + +**Additional context** +Add any other context or screenshots about the feature request here. From 14f0b21ecb7775e2ba5772d8e64525a438c0b1bd Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 24 May 2018 13:50:59 +0200 Subject: [PATCH 03/34] Added contribution guide --- CONTRIBUTING.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..0794a56 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,22 @@ +# Contribution Guide + +We follow the GitHub [Fork & Pull][forkandpull] and [Git Flow][gitflow] workflow in this project. +Please fork the official repository, develop in a branch based on the +current develop branch, and submit a pull request (PR) after you are done. + +[forkandpull]: https://help.github.com/articles/about-pull-requests/ +[gitflow]: http://nvie.com/posts/a-successful-git-branching-model/ + +## Checklist before submitting a Pull Request + +Before you submit your PR, please go through this list and check if +your request fulfills these points. + +- Do you have more changes/additions/deletions in your PR than you expected? +- Do you have tests for your changes? +- Do all tests of the project succeed? +- Is your pull request based on the `develop` branch? [(cf. GitFlow)][gitflow] + +If you check these items before, your pull request is more likely to be +included into the project quickly. + From e3cd594fc16fd3964fd0e69824b4f40399c008cd Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 24 May 2018 16:38:17 +0200 Subject: [PATCH 04/34] Adding elastic search dependency --- build.sbt | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/build.sbt b/build.sbt index a123d92..e418ad3 100644 --- a/build.sbt +++ b/build.sbt @@ -7,6 +7,22 @@ scalaVersion := "2.12.4" libraryDependencies += "org.parboiled" %% "parboiled" % "2.1.4" libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.11" +val elastic4sVersion = "6.2.8" +libraryDependencies ++= Seq( + "com.sksamuel.elastic4s" %% "elastic4s-core" % elastic4sVersion, + + // for the http client + "com.sksamuel.elastic4s" %% "elastic4s-http" % elastic4sVersion, + + // if you want to use reactive streams + "com.sksamuel.elastic4s" %% "elastic4s-http-streams" % elastic4sVersion, + + // testing + "com.sksamuel.elastic4s" %% "elastic4s-testkit" % elastic4sVersion % "test", + "com.sksamuel.elastic4s" %% "elastic4s-embedded" % elastic4sVersion % "test" +) + + lazy val webapi = (project in file(".")). enablePlugins(JavaAppPackaging). enablePlugins(DockerPlugin). From 4cb22465111679978d05dd861d171cd588159b91 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 24 May 2018 16:38:31 +0200 Subject: [PATCH 05/34] Added a configuration object --- .../de/upb/cs/swt/delphi/webapi/Configuration.scala | 13 +++++++++++++ .../scala/de/upb/cs/swt/delphi/webapi/Server.scala | 3 ++- 2 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala new file mode 100644 index 0000000..f61f6b6 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala @@ -0,0 +1,13 @@ +package de.upb.cs.swt.delphi.webapi + +import com.sksamuel.elastic4s.ElasticsearchClientUri + +/** + * @author Ben Hermann + */ +class Configuration(val bindHost: String = "0.0.0.0", + val bindPort: Int = 8080, + val elasticsearchClientUri: ElasticsearchClientUri = ElasticsearchClientUri( + sys.env.getOrElse("DELPHI_ELASTIC_URI", "elasticsearch://localhost:9200"))) { + +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 79e4e64..7a7c757 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -45,7 +45,8 @@ object Server extends HttpApp { } def main(args: Array[String]): Unit = { - Server.startServer("0.0.0.0", 8080) + val configuration = new Configuration() + Server.startServer(configuration.bindHost, configuration.bindPort) } From 447233b0326ba4138397e26bf18528301f47ee26 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan-Niclas=20Str=C3=BCwer?= Date: Mon, 28 May 2018 10:41:22 +0200 Subject: [PATCH 06/34] updated sbt version to 1.1.1 to match the version of the main repository --- project/build.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project/build.properties b/project/build.properties index cd7364c..9f782f7 100644 --- a/project/build.properties +++ b/project/build.properties @@ -1 +1 @@ -sbt.version = 0.13.17 \ No newline at end of file +sbt.version=1.1.1 \ No newline at end of file From 33d634940308256b17287ad987ff7097769ffbec Mon Sep 17 00:00:00 2001 From: almacken Date: Mon, 28 May 2018 16:14:13 -0600 Subject: [PATCH 07/34] Added feature endpoint. Note that it requires features to be mapped to feature descriptions to return them, and currently no features are mapped so no features are returned. --- build.sbt | 3 + .../FeatureDescription.scala | 4 + .../FeatureListMapping.scala | 404 ++++++++++++++++++ .../cs/swt/delphi/webapi/JsonSupport.scala | 6 + .../de/upb/cs/swt/delphi/webapi/Server.scala | 6 +- 5 files changed, 421 insertions(+), 2 deletions(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala diff --git a/build.sbt b/build.sbt index e418ad3..9293e5f 100644 --- a/build.sbt +++ b/build.sbt @@ -6,6 +6,9 @@ scalaVersion := "2.12.4" libraryDependencies += "org.parboiled" %% "parboiled" % "2.1.4" libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.11" +libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.12" +libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.1.1" +libraryDependencies += "io.spray" %% "spray-json" % "1.3.3" val elastic4sVersion = "6.2.8" libraryDependencies ++= Seq( diff --git a/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala new file mode 100644 index 0000000..944d03a --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala @@ -0,0 +1,4 @@ +package de.upb.cs.swt.delphi.featuredefinitions + +//Describes all relevant fields for conducting searches on this class +case class FeatureDescription(fType: String) diff --git a/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala new file mode 100644 index 0000000..d4c165f --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala @@ -0,0 +1,404 @@ +package de.upb.cs.swt.delphi.featuredefinitions + +object FeatureListMapping { + + //Returns a list of all defined features + def featureList: List[String] = + (for {(key, des) <- featureMap if des != null} yield key).toList + + //Maps all field names onto descriptions of that field. Null fields cannot be searched in the current version + private val featureMap = Map[String, FeatureDescription]( + "ProjectPackages" -> null, + "⟨SizeOfInheritanceTree⟩" -> null, + "ProjectFields" -> null, + "LibraryMethods" -> null, + "ProjectInstructions" -> null, + "ProjectMethods" -> null, + "LibraryClassFiles" -> null, + "LibraryFields" -> null, + "LibraryPackages" -> null, + "ProjectClassFiles" -> null, + "0 FPC" -> null, + "1-3 FPC" -> null, + "4-10 FPC" -> null, + ">10 FPC" -> null, + "0 MPC" -> null, + "1-3 MPC" -> null, + "4-10 MPC" -> null, + ">10 MPC" -> null, + "1-3 CPP" -> null, + "4-10 CPP" -> null, + ">10 CPP" -> null, + "0 NOC" -> null, + "1-3 NOC" -> null, + "4-10 NOC" -> null, + ">10 NOC" -> null, + "linear methods (McCabe)" -> null, + "2-3 McCabe" -> null, + "4-10 McCabe" -> null, + ">10 McCabe" -> null, + "Designator" -> null, + "Taxonomy" -> null, + "Joiner" -> null, + "Pool" -> null, + "Function Pointer" -> null, + "Function Object" -> null, + "Cobol Like" -> null, + "Stateless" -> null, + "Common State" -> null, + "Immutable" -> null, + "Restricted Creation" -> null, + "Sampler" -> null, + "Box" -> null, + "Compound Box" -> null, + "Canopy" -> null, + "Record" -> null, + "Data Manager" -> null, + "Sink" -> null, + "Outline" -> null, + "Trait" -> null, + "State Machine" -> null, + "Pure Type" -> null, + "Augmented Type" -> null, + "Pseudo Class" -> null, + "Implementor" -> null, + "Overrider" -> null, + "Extender" -> null, + "unused private fields" -> null, + "unused package visible fields" -> null, + "unused protected fields" -> null, + "unused public fields" -> null, + "package visible fields only used by defining type" -> null, + "protected fields only used by defining type" -> null, + "public fields only used by defininig type " -> null, + "Trivial Class.forName Usage" -> null, + "Nontrivial Class.forName Usage" -> null, + "nop (opcode:0)" -> null, + "aconst_null (opcode:1)" -> null, + "iconst_m1 (opcode:2)" -> null, + "iconst_0 (opcode:3)" -> null, + "iconst_1 (opcode:4)" -> null, + "iconst_2 (opcode:5)" -> null, + "iconst_3 (opcode:6)" -> null, + "iconst_4 (opcode:7)" -> null, + "iconst_5 (opcode:8)" -> null, + "lconst_0 (opcode:9)" -> null, + "lconst_1 (opcode:10)" -> null, + "fconst_0 (opcode:11)" -> null, + "fconst_1 (opcode:12)" -> null, + "fconst_2 (opcode:13)" -> null, + "dconst_0 (opcode:14)" -> null, + "dconst_1 (opcode:15)" -> null, + "bipush (opcode:16)" -> null, + "sipush (opcode:17)" -> null, + "ldc (opcode:18)" -> null, + "ldc_w (opcode:19)" -> null, + "ldc2_w (opcode:20)" -> null, + "iload (opcode:21)" -> null, + "lload (opcode:22)" -> null, + "fload (opcode:23)" -> null, + "dload (opcode:24)" -> null, + "aload (opcode:25)" -> null, + "iload_0 (opcode:26)" -> null, + "iload_1 (opcode:27)" -> null, + "iload_2 (opcode:28)" -> null, + "iload_3 (opcode:29)" -> null, + "lload_0 (opcode:30)" -> null, + "lload_1 (opcode:31)" -> null, + "lload_2 (opcode:32)" -> null, + "lload_3 (opcode:33)" -> null, + "fload_0 (opcode:34)" -> null, + "fload_1 (opcode:35)" -> null, + "fload_2 (opcode:36)" -> null, + "fload_3 (opcode:37)" -> null, + "dload_0 (opcode:38)" -> null, + "dload_1 (opcode:39)" -> null, + "dload_2 (opcode:40)" -> null, + "dload_3 (opcode:41)" -> null, + "aload_0 (opcode:42)" -> null, + "aload_1 (opcode:43)" -> null, + "aload_2 (opcode:44)" -> null, + "aload_3 (opcode:45)" -> null, + "iaload (opcode:46)" -> null, + "laload (opcode:47)" -> null, + "faload (opcode:48)" -> null, + "daload (opcode:49)" -> null, + "aaload (opcode:50)" -> null, + "baload (opcode:51)" -> null, + "caload (opcode:52)" -> null, + "saload (opcode:53)" -> null, + "istore (opcode:54)" -> null, + "lstore (opcode:55)" -> null, + "fstore (opcode:56)" -> null, + "dstore (opcode:57)" -> null, + "astore (opcode:58)" -> null, + "istore_0 (opcode:59)" -> null, + "istore_1 (opcode:60)" -> null, + "istore_2 (opcode:61)" -> null, + "istore_3 (opcode:62)" -> null, + "lstore_0 (opcode:63)" -> null, + "lstore_1 (opcode:64)" -> null, + "lstore_2 (opcode:65)" -> null, + "lstore_3 (opcode:66)" -> null, + "fstore_0 (opcode:67)" -> null, + "fstore_1 (opcode:68)" -> null, + "fstore_2 (opcode:69)" -> null, + "fstore_3 (opcode:70)" -> null, + "dstore_0 (opcode:71)" -> null, + "dstore_1 (opcode:72)" -> null, + "dstore_2 (opcode:73)" -> null, + "dstore_3 (opcode:74)" -> null, + "astore_0 (opcode:75)" -> null, + "astore_1 (opcode:76)" -> null, + "astore_2 (opcode:77)" -> null, + "astore_3 (opcode:78)" -> null, + "iastore (opcode:79)" -> null, + "lastore (opcode:80)" -> null, + "fastore (opcode:81)" -> null, + "dastore (opcode:82)" -> null, + "aastore (opcode:83)" -> null, + "bastore (opcode:84)" -> null, + "castore (opcode:85)" -> null, + "sastore (opcode:86)" -> null, + "pop (opcode:87)" -> null, + "pop2 (opcode:88)" -> null, + "dup (opcode:89)" -> null, + "dup_x1 (opcode:90)" -> null, + "dup_x2 (opcode:91)" -> null, + "dup2 (opcode:92)" -> null, + "dup2_x1 (opcode:93)" -> null, + "dup2_x2 (opcode:94)" -> null, + "swap (opcode:95)" -> null, + "iadd (opcode:96)" -> null, + "ladd (opcode:97)" -> null, + "fadd (opcode:98)" -> null, + "dadd (opcode:99)" -> null, + "isub (opcode:100)" -> null, + "lsub (opcode:101)" -> null, + "fsub (opcode:102)" -> null, + "dsub (opcode:103)" -> null, + "imul (opcode:104)" -> null, + "lmul (opcode:105)" -> null, + "fmul (opcode:106)" -> null, + "dmul (opcode:107)" -> null, + "idiv (opcode:108)" -> null, + "ldiv (opcode:109)" -> null, + "fdiv (opcode:110)" -> null, + "ddiv (opcode:111)" -> null, + "irem (opcode:112)" -> null, + "lrem (opcode:113)" -> null, + "frem (opcode:114)" -> null, + "drem (opcode:115)" -> null, + "ineg (opcode:116)" -> null, + "lneg (opcode:117)" -> null, + "fneg (opcode:118)" -> null, + "dneg (opcode:119)" -> null, + "ishl (opcode:120)" -> null, + "lshl (opcode:121)" -> null, + "ishr (opcode:122)" -> null, + "lshr (opcode:123)" -> null, + "iushr (opcode:124)" -> null, + "lushr (opcode:125)" -> null, + "iand (opcode:126)" -> null, + "land (opcode:127)" -> null, + "ior (opcode:128)" -> null, + "lor (opcode:129)" -> null, + "ixor (opcode:130)" -> null, + "lxor (opcode:131)" -> null, + "iinc (opcode:132)" -> null, + "i2l (opcode:133)" -> null, + "i2f (opcode:134)" -> null, + "i2d (opcode:135)" -> null, + "l2i (opcode:136)" -> null, + "l2f (opcode:137)" -> null, + "l2d (opcode:138)" -> null, + "f2i (opcode:139)" -> null, + "f2l (opcode:140)" -> null, + "f2d (opcode:141)" -> null, + "d2i (opcode:142)" -> null, + "d2l (opcode:143)" -> null, + "d2f (opcode:144)" -> null, + "i2b (opcode:145)" -> null, + "i2c (opcode:146)" -> null, + "i2s (opcode:147)" -> null, + "lcmp (opcode:148)" -> null, + "fcmpl (opcode:149)" -> null, + "fcmpg (opcode:150)" -> null, + "dcmpl (opcode:151)" -> null, + "dcmpg (opcode:152)" -> null, + "ifeq (opcode:153)" -> null, + "ifne (opcode:154)" -> null, + "iflt (opcode:155)" -> null, + "ifge (opcode:156)" -> null, + "ifgt (opcode:157)" -> null, + "ifle (opcode:158)" -> null, + "if_icmpeq (opcode:159)" -> null, + "if_icmpne (opcode:160)" -> null, + "if_icmplt (opcode:161)" -> null, + "if_icmpge (opcode:162)" -> null, + "if_icmpgt (opcode:163)" -> null, + "if_icmple (opcode:164)" -> null, + "if_acmpeq (opcode:165)" -> null, + "if_acmpne (opcode:166)" -> null, + "goto (opcode:167)" -> null, + "jsr (opcode:168)" -> null, + "ret (opcode:169)" -> null, + "tableswitch (opcode:170)" -> null, + "lookupswitch (opcode:171)" -> null, + "ireturn (opcode:172)" -> null, + "lreturn (opcode:173)" -> null, + "freturn (opcode:174)" -> null, + "dreturn (opcode:175)" -> null, + "areturn (opcode:176)" -> null, + "return (opcode:177)" -> null, + "getstatic (opcode:178)" -> null, + "putstatic (opcode:179)" -> null, + "getfield (opcode:180)" -> null, + "putfield (opcode:181)" -> null, + "invokevirtual (opcode:182)" -> null, + "invokespecial (opcode:183)" -> null, + "invokestatic (opcode:184)" -> null, + "invokeinterface (opcode:185)" -> null, + "invokedynamic (opcode:186)" -> null, + "new (opcode:187)" -> null, + "newarray (opcode:188)" -> null, + "anewarray (opcode:189)" -> null, + "arraylength (opcode:190)" -> null, + "athrow (opcode:191)" -> null, + "checkcast (opcode:192)" -> null, + "instanceof (opcode:193)" -> null, + "monitorenter (opcode:194)" -> null, + "monitorexit (opcode:195)" -> null, + "wide (opcode:196)" -> null, + "multianewarray (opcode:197)" -> null, + "ifnull (opcode:198)" -> null, + "ifnonnull (opcode:199)" -> null, + "goto_w (opcode:200)" -> null, + "jsr_w (opcode:201)" -> null, + "Self-recursive Data Structure" -> null, + "Mutually-recursive Data Structure 2 Types" -> null, + "Mutually-recursive Data Structure 3 Types" -> null, + "Mutually-recursive Data Structure 4 Types" -> null, + "Mutually-recursive Data Structure more than 4 Types" -> null, + "Never Returns Normally" -> null, + "Method with Infinite Loop" -> null, + "Class File With Source Attribute" -> null, + "Method With Line Number Table" -> null, + "Method With Local Variable Table" -> null, + "Method With Local Variable Type Table" -> null, + "FanOut - Category 1" -> null, + "FanOut - Category 2" -> null, + "FanOut - Category 3" -> null, + "FanOut - Category 4" -> null, + "FanOut - Category 5" -> null, + "FanOut - Category 6" -> null, + "FanIn - Category 1" -> null, + "FanIn - Category 2" -> null, + "FanIn - Category 3" -> null, + "FanIn - Category 4" -> null, + "FanIn - Category 5" -> null, + "FanIn - Category 6" -> null, + "FanIn/FanOut - Category 1" -> null, + "FanIn/FanOut - Category 2" -> null, + "FanIn/FanOut - Category 3" -> null, + "FanIn/FanOut - Category 4" -> null, + "FanIn/FanOut - Category 5" -> null, + "FanIn/FanOut - Category 6" -> null, + "custom ClassLoader implementation" -> null, + "Retrieving the SystemClassLoader" -> null, + "Retrieving some ClassLoader" -> null, + "define new classes/packages" -> null, + "accessing resources" -> null, + "javax.crypto.Cipher getInstance" -> null, + "using SecureRandom" -> null, + "using MessageDigest" -> null, + "using Signature" -> null, + "using Mac" -> null, + "cryptographic key handling" -> null, + "using KeyStore" -> null, + "using Certificates" -> null, + "native methods" -> null, + "synthetic methods" -> null, + "bridge methods" -> null, + "synchronized methods" -> null, + "varargs methods" -> null, + "static initializers" -> null, + "static methods (not including static initializers)" -> null, + "constructors" -> null, + "instance methods" -> null, + "java.lang.Class forName" -> null, + "reflective instance creation" -> null, + "reflective field write" -> null, + "reflective field read" -> null, + "makes fields accessible" -> null, + "makes methods or constructors accessible" -> null, + "makes an AccessibleObject accessible (exact type unknown)" -> null, + "java.lang.reflect.Method Object invoke(Object, Object[])" -> null, + "java.lang.invoke.MethodHandles lookup" -> null, + "java.lang.invoke.MethodHandles publicLookup" -> null, + "method handle invocation" -> null, + "java.lang.reflect.Proxy newProxyInstance" -> null, + "Process" -> null, + "JVM exit" -> null, + "Native Libraries" -> null, + "java.lang.System getSecurityManager" -> null, + "java.lang.System setSecurityManager" -> null, + "Environment" -> null, + "Sound" -> null, + "Network sockets" -> null, + "Object-based Thread Notification" -> null, + "Usage of Thread API" -> null, + "Usage of ThreadGroup API" -> null, + "sun.misc.Unsafe sun.misc.Unsafe getUnsafe()" -> null, + "Unsafe - Alloc" -> null, + "Unsafe - Array" -> null, + "Unsafe - compareAndSwap" -> null, + "Unsafe - Class" -> null, + "Unsafe - Fence" -> null, + "Unsafe - Fetch & Add" -> null, + "Unsafe - Heap" -> null, + "Unsafe - Heap Get" -> null, + "Unsafe - Heap Put" -> null, + "Misc" -> null, + "Unsafe - Monitor" -> null, + "Unsafe - Off-Heap" -> null, + "Unsafe - Offset" -> null, + "Unsafe - Ordered Put" -> null, + "Unsafe - Park" -> null, + "Unsafe - Throw" -> null, + "Unsafe - Volatile Get" -> null, + "Unsafe - Volatile Put" -> null, + "java.sql.DriverManager getConnection" -> null, + "java.sql.Connection rollback" -> null, + "creation and execution of java.sql.Statement" -> null, + "creation and execution of java.sql.PreparedStatement" -> null, + "creation and execution of java.sql.CallableStatement" -> null, + "class file retransformation" -> null, + "instrumenting native methods" -> null, + "appending class loader search" -> null, + "retrieve classes information" -> null, + "(concrete) classes" -> null, + "abstract classes" -> null, + "annotations" -> null, + "enumerations" -> null, + "marker interfaces" -> null, + "simple functional interfaces (single abstract method (SAM) interface)" -> null, + "non-functional interface with default methods (Java >8)" -> null, + "non-functional interface with static methods (Java >8)" -> null, + "(standard) interface" -> null, + "module (Java >9)" -> null, + "Very Small Inheritance Tree" -> null, + "Small Inheritance Tree" -> null, + "Medium Inheritance Tree" -> null, + "Large Inheritance Tree" -> null, + "Very Large Inheritance Tree" -> null, + "Huge Inheritance Tree" -> null, + "Size of the Inheritance Tree Unknown" -> null, + "Class File JDK 1.1 (JDK 1.0.2)" -> null, + "Class File Java 5" -> null, + "Class File Java 6" -> null, + "Class File Java 7" -> null, + "Class File Java 8" -> null, + "Class File Java 9" -> null + ) +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala new file mode 100644 index 0000000..4d8c771 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala @@ -0,0 +1,6 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport +import spray.json._ + +trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 7a7c757..36dd6f0 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -1,11 +1,13 @@ package de.upb.cs.swt.delphi.webapi import akka.http.scaladsl.server.HttpApp +import de.upb.cs.swt.delphi.featuredefinitions.FeatureListMapping +import spray.json._ /** * Web server configuration for Delphi web API. */ -object Server extends HttpApp { +object Server extends HttpApp with JsonSupport { override def routes = path("version") { version } ~ @@ -25,7 +27,7 @@ object Server extends HttpApp { private def features = { get { complete { - "features" + FeatureListMapping.featureList.toJson } } } From ea3bd8599b84e1934ff42afa256286acce7d808f Mon Sep 17 00:00:00 2001 From: almacken Date: Tue, 29 May 2018 12:33:16 -0600 Subject: [PATCH 08/34] Added retrieve endpoint --- .../cs/swt/delphi/webapi/ElasticClient.scala | 20 +++++++++++++++++++ .../de/upb/cs/swt/delphi/webapi/Server.scala | 4 +++- 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala new file mode 100644 index 0000000..8e12d13 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala @@ -0,0 +1,20 @@ +package de.upb.cs.swt.delphi.webapi + +import com.sksamuel.elastic4s.http.ElasticDsl._ +import com.sksamuel.elastic4s.http.HttpClient + +object ElasticClient { + + val configuration = new Configuration() + val client = HttpClient(configuration.elasticsearchClientUri) + val index = "delphi" / "project" + + //Returns an entry with the given ID as an option + def getSource(id: String) = + client.execute{ + get(id).from(index) + }.await match { + case Right(res) => res.body + case Left(_) => Option.empty + } +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 36dd6f0..16926b4 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -34,7 +34,9 @@ object Server extends HttpApp with JsonSupport { def retrieve(identifier: String) = { get { - complete(identifier) + complete( + ElasticClient.getSource(identifier) + ) } } From 8d2997ea79392050d639fc56866aa8e9dceb9843 Mon Sep 17 00:00:00 2001 From: almacken Date: Mon, 28 May 2018 16:14:13 -0600 Subject: [PATCH 09/34] Added feature endpoint. Note that it requires features to be mapped to feature descriptions to return them, and currently no features are mapped so no features are returned. --- build.sbt | 3 + .../FeatureDescription.scala | 4 + .../FeatureListMapping.scala | 404 ++++++++++++++++++ .../cs/swt/delphi/webapi/JsonSupport.scala | 6 + .../de/upb/cs/swt/delphi/webapi/Server.scala | 6 +- 5 files changed, 421 insertions(+), 2 deletions(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala diff --git a/build.sbt b/build.sbt index e418ad3..9293e5f 100644 --- a/build.sbt +++ b/build.sbt @@ -6,6 +6,9 @@ scalaVersion := "2.12.4" libraryDependencies += "org.parboiled" %% "parboiled" % "2.1.4" libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.11" +libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.12" +libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.1.1" +libraryDependencies += "io.spray" %% "spray-json" % "1.3.3" val elastic4sVersion = "6.2.8" libraryDependencies ++= Seq( diff --git a/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala new file mode 100644 index 0000000..944d03a --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureDescription.scala @@ -0,0 +1,4 @@ +package de.upb.cs.swt.delphi.featuredefinitions + +//Describes all relevant fields for conducting searches on this class +case class FeatureDescription(fType: String) diff --git a/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala new file mode 100644 index 0000000..d4c165f --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/featuredefinitions/FeatureListMapping.scala @@ -0,0 +1,404 @@ +package de.upb.cs.swt.delphi.featuredefinitions + +object FeatureListMapping { + + //Returns a list of all defined features + def featureList: List[String] = + (for {(key, des) <- featureMap if des != null} yield key).toList + + //Maps all field names onto descriptions of that field. Null fields cannot be searched in the current version + private val featureMap = Map[String, FeatureDescription]( + "ProjectPackages" -> null, + "⟨SizeOfInheritanceTree⟩" -> null, + "ProjectFields" -> null, + "LibraryMethods" -> null, + "ProjectInstructions" -> null, + "ProjectMethods" -> null, + "LibraryClassFiles" -> null, + "LibraryFields" -> null, + "LibraryPackages" -> null, + "ProjectClassFiles" -> null, + "0 FPC" -> null, + "1-3 FPC" -> null, + "4-10 FPC" -> null, + ">10 FPC" -> null, + "0 MPC" -> null, + "1-3 MPC" -> null, + "4-10 MPC" -> null, + ">10 MPC" -> null, + "1-3 CPP" -> null, + "4-10 CPP" -> null, + ">10 CPP" -> null, + "0 NOC" -> null, + "1-3 NOC" -> null, + "4-10 NOC" -> null, + ">10 NOC" -> null, + "linear methods (McCabe)" -> null, + "2-3 McCabe" -> null, + "4-10 McCabe" -> null, + ">10 McCabe" -> null, + "Designator" -> null, + "Taxonomy" -> null, + "Joiner" -> null, + "Pool" -> null, + "Function Pointer" -> null, + "Function Object" -> null, + "Cobol Like" -> null, + "Stateless" -> null, + "Common State" -> null, + "Immutable" -> null, + "Restricted Creation" -> null, + "Sampler" -> null, + "Box" -> null, + "Compound Box" -> null, + "Canopy" -> null, + "Record" -> null, + "Data Manager" -> null, + "Sink" -> null, + "Outline" -> null, + "Trait" -> null, + "State Machine" -> null, + "Pure Type" -> null, + "Augmented Type" -> null, + "Pseudo Class" -> null, + "Implementor" -> null, + "Overrider" -> null, + "Extender" -> null, + "unused private fields" -> null, + "unused package visible fields" -> null, + "unused protected fields" -> null, + "unused public fields" -> null, + "package visible fields only used by defining type" -> null, + "protected fields only used by defining type" -> null, + "public fields only used by defininig type " -> null, + "Trivial Class.forName Usage" -> null, + "Nontrivial Class.forName Usage" -> null, + "nop (opcode:0)" -> null, + "aconst_null (opcode:1)" -> null, + "iconst_m1 (opcode:2)" -> null, + "iconst_0 (opcode:3)" -> null, + "iconst_1 (opcode:4)" -> null, + "iconst_2 (opcode:5)" -> null, + "iconst_3 (opcode:6)" -> null, + "iconst_4 (opcode:7)" -> null, + "iconst_5 (opcode:8)" -> null, + "lconst_0 (opcode:9)" -> null, + "lconst_1 (opcode:10)" -> null, + "fconst_0 (opcode:11)" -> null, + "fconst_1 (opcode:12)" -> null, + "fconst_2 (opcode:13)" -> null, + "dconst_0 (opcode:14)" -> null, + "dconst_1 (opcode:15)" -> null, + "bipush (opcode:16)" -> null, + "sipush (opcode:17)" -> null, + "ldc (opcode:18)" -> null, + "ldc_w (opcode:19)" -> null, + "ldc2_w (opcode:20)" -> null, + "iload (opcode:21)" -> null, + "lload (opcode:22)" -> null, + "fload (opcode:23)" -> null, + "dload (opcode:24)" -> null, + "aload (opcode:25)" -> null, + "iload_0 (opcode:26)" -> null, + "iload_1 (opcode:27)" -> null, + "iload_2 (opcode:28)" -> null, + "iload_3 (opcode:29)" -> null, + "lload_0 (opcode:30)" -> null, + "lload_1 (opcode:31)" -> null, + "lload_2 (opcode:32)" -> null, + "lload_3 (opcode:33)" -> null, + "fload_0 (opcode:34)" -> null, + "fload_1 (opcode:35)" -> null, + "fload_2 (opcode:36)" -> null, + "fload_3 (opcode:37)" -> null, + "dload_0 (opcode:38)" -> null, + "dload_1 (opcode:39)" -> null, + "dload_2 (opcode:40)" -> null, + "dload_3 (opcode:41)" -> null, + "aload_0 (opcode:42)" -> null, + "aload_1 (opcode:43)" -> null, + "aload_2 (opcode:44)" -> null, + "aload_3 (opcode:45)" -> null, + "iaload (opcode:46)" -> null, + "laload (opcode:47)" -> null, + "faload (opcode:48)" -> null, + "daload (opcode:49)" -> null, + "aaload (opcode:50)" -> null, + "baload (opcode:51)" -> null, + "caload (opcode:52)" -> null, + "saload (opcode:53)" -> null, + "istore (opcode:54)" -> null, + "lstore (opcode:55)" -> null, + "fstore (opcode:56)" -> null, + "dstore (opcode:57)" -> null, + "astore (opcode:58)" -> null, + "istore_0 (opcode:59)" -> null, + "istore_1 (opcode:60)" -> null, + "istore_2 (opcode:61)" -> null, + "istore_3 (opcode:62)" -> null, + "lstore_0 (opcode:63)" -> null, + "lstore_1 (opcode:64)" -> null, + "lstore_2 (opcode:65)" -> null, + "lstore_3 (opcode:66)" -> null, + "fstore_0 (opcode:67)" -> null, + "fstore_1 (opcode:68)" -> null, + "fstore_2 (opcode:69)" -> null, + "fstore_3 (opcode:70)" -> null, + "dstore_0 (opcode:71)" -> null, + "dstore_1 (opcode:72)" -> null, + "dstore_2 (opcode:73)" -> null, + "dstore_3 (opcode:74)" -> null, + "astore_0 (opcode:75)" -> null, + "astore_1 (opcode:76)" -> null, + "astore_2 (opcode:77)" -> null, + "astore_3 (opcode:78)" -> null, + "iastore (opcode:79)" -> null, + "lastore (opcode:80)" -> null, + "fastore (opcode:81)" -> null, + "dastore (opcode:82)" -> null, + "aastore (opcode:83)" -> null, + "bastore (opcode:84)" -> null, + "castore (opcode:85)" -> null, + "sastore (opcode:86)" -> null, + "pop (opcode:87)" -> null, + "pop2 (opcode:88)" -> null, + "dup (opcode:89)" -> null, + "dup_x1 (opcode:90)" -> null, + "dup_x2 (opcode:91)" -> null, + "dup2 (opcode:92)" -> null, + "dup2_x1 (opcode:93)" -> null, + "dup2_x2 (opcode:94)" -> null, + "swap (opcode:95)" -> null, + "iadd (opcode:96)" -> null, + "ladd (opcode:97)" -> null, + "fadd (opcode:98)" -> null, + "dadd (opcode:99)" -> null, + "isub (opcode:100)" -> null, + "lsub (opcode:101)" -> null, + "fsub (opcode:102)" -> null, + "dsub (opcode:103)" -> null, + "imul (opcode:104)" -> null, + "lmul (opcode:105)" -> null, + "fmul (opcode:106)" -> null, + "dmul (opcode:107)" -> null, + "idiv (opcode:108)" -> null, + "ldiv (opcode:109)" -> null, + "fdiv (opcode:110)" -> null, + "ddiv (opcode:111)" -> null, + "irem (opcode:112)" -> null, + "lrem (opcode:113)" -> null, + "frem (opcode:114)" -> null, + "drem (opcode:115)" -> null, + "ineg (opcode:116)" -> null, + "lneg (opcode:117)" -> null, + "fneg (opcode:118)" -> null, + "dneg (opcode:119)" -> null, + "ishl (opcode:120)" -> null, + "lshl (opcode:121)" -> null, + "ishr (opcode:122)" -> null, + "lshr (opcode:123)" -> null, + "iushr (opcode:124)" -> null, + "lushr (opcode:125)" -> null, + "iand (opcode:126)" -> null, + "land (opcode:127)" -> null, + "ior (opcode:128)" -> null, + "lor (opcode:129)" -> null, + "ixor (opcode:130)" -> null, + "lxor (opcode:131)" -> null, + "iinc (opcode:132)" -> null, + "i2l (opcode:133)" -> null, + "i2f (opcode:134)" -> null, + "i2d (opcode:135)" -> null, + "l2i (opcode:136)" -> null, + "l2f (opcode:137)" -> null, + "l2d (opcode:138)" -> null, + "f2i (opcode:139)" -> null, + "f2l (opcode:140)" -> null, + "f2d (opcode:141)" -> null, + "d2i (opcode:142)" -> null, + "d2l (opcode:143)" -> null, + "d2f (opcode:144)" -> null, + "i2b (opcode:145)" -> null, + "i2c (opcode:146)" -> null, + "i2s (opcode:147)" -> null, + "lcmp (opcode:148)" -> null, + "fcmpl (opcode:149)" -> null, + "fcmpg (opcode:150)" -> null, + "dcmpl (opcode:151)" -> null, + "dcmpg (opcode:152)" -> null, + "ifeq (opcode:153)" -> null, + "ifne (opcode:154)" -> null, + "iflt (opcode:155)" -> null, + "ifge (opcode:156)" -> null, + "ifgt (opcode:157)" -> null, + "ifle (opcode:158)" -> null, + "if_icmpeq (opcode:159)" -> null, + "if_icmpne (opcode:160)" -> null, + "if_icmplt (opcode:161)" -> null, + "if_icmpge (opcode:162)" -> null, + "if_icmpgt (opcode:163)" -> null, + "if_icmple (opcode:164)" -> null, + "if_acmpeq (opcode:165)" -> null, + "if_acmpne (opcode:166)" -> null, + "goto (opcode:167)" -> null, + "jsr (opcode:168)" -> null, + "ret (opcode:169)" -> null, + "tableswitch (opcode:170)" -> null, + "lookupswitch (opcode:171)" -> null, + "ireturn (opcode:172)" -> null, + "lreturn (opcode:173)" -> null, + "freturn (opcode:174)" -> null, + "dreturn (opcode:175)" -> null, + "areturn (opcode:176)" -> null, + "return (opcode:177)" -> null, + "getstatic (opcode:178)" -> null, + "putstatic (opcode:179)" -> null, + "getfield (opcode:180)" -> null, + "putfield (opcode:181)" -> null, + "invokevirtual (opcode:182)" -> null, + "invokespecial (opcode:183)" -> null, + "invokestatic (opcode:184)" -> null, + "invokeinterface (opcode:185)" -> null, + "invokedynamic (opcode:186)" -> null, + "new (opcode:187)" -> null, + "newarray (opcode:188)" -> null, + "anewarray (opcode:189)" -> null, + "arraylength (opcode:190)" -> null, + "athrow (opcode:191)" -> null, + "checkcast (opcode:192)" -> null, + "instanceof (opcode:193)" -> null, + "monitorenter (opcode:194)" -> null, + "monitorexit (opcode:195)" -> null, + "wide (opcode:196)" -> null, + "multianewarray (opcode:197)" -> null, + "ifnull (opcode:198)" -> null, + "ifnonnull (opcode:199)" -> null, + "goto_w (opcode:200)" -> null, + "jsr_w (opcode:201)" -> null, + "Self-recursive Data Structure" -> null, + "Mutually-recursive Data Structure 2 Types" -> null, + "Mutually-recursive Data Structure 3 Types" -> null, + "Mutually-recursive Data Structure 4 Types" -> null, + "Mutually-recursive Data Structure more than 4 Types" -> null, + "Never Returns Normally" -> null, + "Method with Infinite Loop" -> null, + "Class File With Source Attribute" -> null, + "Method With Line Number Table" -> null, + "Method With Local Variable Table" -> null, + "Method With Local Variable Type Table" -> null, + "FanOut - Category 1" -> null, + "FanOut - Category 2" -> null, + "FanOut - Category 3" -> null, + "FanOut - Category 4" -> null, + "FanOut - Category 5" -> null, + "FanOut - Category 6" -> null, + "FanIn - Category 1" -> null, + "FanIn - Category 2" -> null, + "FanIn - Category 3" -> null, + "FanIn - Category 4" -> null, + "FanIn - Category 5" -> null, + "FanIn - Category 6" -> null, + "FanIn/FanOut - Category 1" -> null, + "FanIn/FanOut - Category 2" -> null, + "FanIn/FanOut - Category 3" -> null, + "FanIn/FanOut - Category 4" -> null, + "FanIn/FanOut - Category 5" -> null, + "FanIn/FanOut - Category 6" -> null, + "custom ClassLoader implementation" -> null, + "Retrieving the SystemClassLoader" -> null, + "Retrieving some ClassLoader" -> null, + "define new classes/packages" -> null, + "accessing resources" -> null, + "javax.crypto.Cipher getInstance" -> null, + "using SecureRandom" -> null, + "using MessageDigest" -> null, + "using Signature" -> null, + "using Mac" -> null, + "cryptographic key handling" -> null, + "using KeyStore" -> null, + "using Certificates" -> null, + "native methods" -> null, + "synthetic methods" -> null, + "bridge methods" -> null, + "synchronized methods" -> null, + "varargs methods" -> null, + "static initializers" -> null, + "static methods (not including static initializers)" -> null, + "constructors" -> null, + "instance methods" -> null, + "java.lang.Class forName" -> null, + "reflective instance creation" -> null, + "reflective field write" -> null, + "reflective field read" -> null, + "makes fields accessible" -> null, + "makes methods or constructors accessible" -> null, + "makes an AccessibleObject accessible (exact type unknown)" -> null, + "java.lang.reflect.Method Object invoke(Object, Object[])" -> null, + "java.lang.invoke.MethodHandles lookup" -> null, + "java.lang.invoke.MethodHandles publicLookup" -> null, + "method handle invocation" -> null, + "java.lang.reflect.Proxy newProxyInstance" -> null, + "Process" -> null, + "JVM exit" -> null, + "Native Libraries" -> null, + "java.lang.System getSecurityManager" -> null, + "java.lang.System setSecurityManager" -> null, + "Environment" -> null, + "Sound" -> null, + "Network sockets" -> null, + "Object-based Thread Notification" -> null, + "Usage of Thread API" -> null, + "Usage of ThreadGroup API" -> null, + "sun.misc.Unsafe sun.misc.Unsafe getUnsafe()" -> null, + "Unsafe - Alloc" -> null, + "Unsafe - Array" -> null, + "Unsafe - compareAndSwap" -> null, + "Unsafe - Class" -> null, + "Unsafe - Fence" -> null, + "Unsafe - Fetch & Add" -> null, + "Unsafe - Heap" -> null, + "Unsafe - Heap Get" -> null, + "Unsafe - Heap Put" -> null, + "Misc" -> null, + "Unsafe - Monitor" -> null, + "Unsafe - Off-Heap" -> null, + "Unsafe - Offset" -> null, + "Unsafe - Ordered Put" -> null, + "Unsafe - Park" -> null, + "Unsafe - Throw" -> null, + "Unsafe - Volatile Get" -> null, + "Unsafe - Volatile Put" -> null, + "java.sql.DriverManager getConnection" -> null, + "java.sql.Connection rollback" -> null, + "creation and execution of java.sql.Statement" -> null, + "creation and execution of java.sql.PreparedStatement" -> null, + "creation and execution of java.sql.CallableStatement" -> null, + "class file retransformation" -> null, + "instrumenting native methods" -> null, + "appending class loader search" -> null, + "retrieve classes information" -> null, + "(concrete) classes" -> null, + "abstract classes" -> null, + "annotations" -> null, + "enumerations" -> null, + "marker interfaces" -> null, + "simple functional interfaces (single abstract method (SAM) interface)" -> null, + "non-functional interface with default methods (Java >8)" -> null, + "non-functional interface with static methods (Java >8)" -> null, + "(standard) interface" -> null, + "module (Java >9)" -> null, + "Very Small Inheritance Tree" -> null, + "Small Inheritance Tree" -> null, + "Medium Inheritance Tree" -> null, + "Large Inheritance Tree" -> null, + "Very Large Inheritance Tree" -> null, + "Huge Inheritance Tree" -> null, + "Size of the Inheritance Tree Unknown" -> null, + "Class File JDK 1.1 (JDK 1.0.2)" -> null, + "Class File Java 5" -> null, + "Class File Java 6" -> null, + "Class File Java 7" -> null, + "Class File Java 8" -> null, + "Class File Java 9" -> null + ) +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala new file mode 100644 index 0000000..4d8c771 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/JsonSupport.scala @@ -0,0 +1,6 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport +import spray.json._ + +trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 7a7c757..36dd6f0 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -1,11 +1,13 @@ package de.upb.cs.swt.delphi.webapi import akka.http.scaladsl.server.HttpApp +import de.upb.cs.swt.delphi.featuredefinitions.FeatureListMapping +import spray.json._ /** * Web server configuration for Delphi web API. */ -object Server extends HttpApp { +object Server extends HttpApp with JsonSupport { override def routes = path("version") { version } ~ @@ -25,7 +27,7 @@ object Server extends HttpApp { private def features = { get { complete { - "features" + FeatureListMapping.featureList.toJson } } } From 4af0304c6d09bb2429cdbc003ede137e4ad92fbf Mon Sep 17 00:00:00 2001 From: almacken Date: Tue, 29 May 2018 12:33:16 -0600 Subject: [PATCH 10/34] Added retrieve endpoint --- .../cs/swt/delphi/webapi/ElasticClient.scala | 20 +++++++++++++++++++ .../de/upb/cs/swt/delphi/webapi/Server.scala | 4 +++- 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala new file mode 100644 index 0000000..8e12d13 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala @@ -0,0 +1,20 @@ +package de.upb.cs.swt.delphi.webapi + +import com.sksamuel.elastic4s.http.ElasticDsl._ +import com.sksamuel.elastic4s.http.HttpClient + +object ElasticClient { + + val configuration = new Configuration() + val client = HttpClient(configuration.elasticsearchClientUri) + val index = "delphi" / "project" + + //Returns an entry with the given ID as an option + def getSource(id: String) = + client.execute{ + get(id).from(index) + }.await match { + case Right(res) => res.body + case Left(_) => Option.empty + } +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 36dd6f0..16926b4 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -34,7 +34,9 @@ object Server extends HttpApp with JsonSupport { def retrieve(identifier: String) = { get { - complete(identifier) + complete( + ElasticClient.getSource(identifier) + ) } } From 769c2b66489a550b67e1a29ccde2c86f39836cfd Mon Sep 17 00:00:00 2001 From: almacken Date: Tue, 5 Jun 2018 16:33:33 -0600 Subject: [PATCH 11/34] Refactored Retrieve to use Actors --- .../cs/swt/delphi/webapi/ElasticActor.scala | 39 +++++++++++++++++++ .../delphi/webapi/ElasticActorManager.scala | 28 +++++++++++++ .../cs/swt/delphi/webapi/ElasticClient.scala | 20 ---------- .../de/upb/cs/swt/delphi/webapi/Server.scala | 14 ++++++- 4 files changed, 80 insertions(+), 21 deletions(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala delete mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala new file mode 100644 index 0000000..1906a77 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -0,0 +1,39 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.actor.{Actor, ActorLogging, PoisonPill, Props, ReceiveTimeout} +import com.sksamuel.elastic4s.IndexAndType +import com.sksamuel.elastic4s.http.ElasticDsl._ +import com.sksamuel.elastic4s.http.HttpClient +import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource + +import scala.concurrent.duration._ + +class ElasticActor(configuration: Configuration) extends Actor with ActorLogging{ + + val client = HttpClient(configuration.elasticsearchClientUri) + + override def preStart(): Unit = log.info("Search actor started") + override def postStop(): Unit = log.info("Search actor shut down") + context.setReceiveTimeout(2 seconds) + + override def receive = { + case GetSource(id, index) => { + log.info("Executing get on entry {}", id) + def source = client.execute{ + get(id).from(index) + }.await match { + case Right(res) => res.body.get + case Left(_) => Option.empty + } + sender().tell(source, context.self) + context.stop(self) + } + case ReceiveTimeout => context.stop(self) + } +} + +object ElasticActor{ + def props(configuration: Configuration) : Props = Props(new ElasticActor(configuration)) + + final case class GetSource(id: String, index: IndexAndType) +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala new file mode 100644 index 0000000..5dbd562 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala @@ -0,0 +1,28 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.actor.{Actor, ActorLogging, Props} +import com.sksamuel.elastic4s.http.ElasticDsl._ +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.Retrieve +import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource + +class ElasticActorManager(configuration: Configuration) extends Actor with ActorLogging{ + + private val index = "delphi" / "project" + + override def preStart(): Unit = log.info("Actor manager started") + override def postStop(): Unit = log.info("Actor manager shut down") + + override def receive = { + case Retrieve(id) => { + log.info("Creating actor to search for entry {}", id) + val retrieveActor = context.actorOf(ElasticActor.props(configuration)) + retrieveActor forward GetSource(id, index) + } + } +} + +object ElasticActorManager{ + def props(configuration: Configuration) : Props = Props(new ElasticActorManager(configuration)) + + final case class Retrieve(id: String) +} \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala deleted file mode 100644 index 8e12d13..0000000 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticClient.scala +++ /dev/null @@ -1,20 +0,0 @@ -package de.upb.cs.swt.delphi.webapi - -import com.sksamuel.elastic4s.http.ElasticDsl._ -import com.sksamuel.elastic4s.http.HttpClient - -object ElasticClient { - - val configuration = new Configuration() - val client = HttpClient(configuration.elasticsearchClientUri) - val index = "delphi" / "project" - - //Returns an entry with the given ID as an option - def getSource(id: String) = - client.execute{ - get(id).from(index) - }.await match { - case Right(res) => res.body - case Left(_) => Option.empty - } -} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 16926b4..a736dd3 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -1,7 +1,13 @@ package de.upb.cs.swt.delphi.webapi +import java.util.concurrent.TimeUnit + +import akka.actor.ActorSystem import akka.http.scaladsl.server.HttpApp +import akka.pattern.ask +import akka.util.Timeout import de.upb.cs.swt.delphi.featuredefinitions.FeatureListMapping +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.Retrieve import spray.json._ /** @@ -9,6 +15,11 @@ import spray.json._ */ object Server extends HttpApp with JsonSupport { + private val configuration = new Configuration() + private val system = ActorSystem("delphi-webapi") + private val actorManager = system.actorOf(ElasticActorManager.props(configuration)) + implicit val timeout = Timeout(5, TimeUnit.SECONDS) + override def routes = path("version") { version } ~ path("features") { features } ~ @@ -35,7 +46,7 @@ object Server extends HttpApp with JsonSupport { def retrieve(identifier: String) = { get { complete( - ElasticClient.getSource(identifier) + (actorManager ? Retrieve(identifier)).mapTo[String] ) } } @@ -51,6 +62,7 @@ object Server extends HttpApp with JsonSupport { def main(args: Array[String]): Unit = { val configuration = new Configuration() Server.startServer(configuration.bindHost, configuration.bindPort) + system.terminate() } From 3f69e794c1b6e37c08469fcac56d293a1684e5be Mon Sep 17 00:00:00 2001 From: almacken Date: Wed, 6 Jun 2018 15:15:29 -0600 Subject: [PATCH 12/34] Added Enqueue route, using a priority mailbox --- src/main/resources/application.conf | 9 +++++++++ .../upb/cs/swt/delphi/webapi/AppLogging.scala | 8 ++++++++ .../upb/cs/swt/delphi/webapi/ElasticActor.scala | 2 +- .../swt/delphi/webapi/ElasticActorManager.scala | 17 +++++++++++------ .../delphi/webapi/ElasticPriorityMailbox.scala | 14 ++++++++++++++ .../de/upb/cs/swt/delphi/webapi/Server.scala | 17 ++++++++++++++--- 6 files changed, 57 insertions(+), 10 deletions(-) create mode 100644 src/main/resources/application.conf create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/AppLogging.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticPriorityMailbox.scala diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf new file mode 100644 index 0000000..9ebf7f5 --- /dev/null +++ b/src/main/resources/application.conf @@ -0,0 +1,9 @@ +es-priority-mailbox { + mailbox-type = "de.upb.cs.swt.delphi.webapi.ElasticPriorityMailbox" +} + +akka.actor.deployment { + /espriomailboxactor { + mailbox = es-priority-mailbox + } +} \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/AppLogging.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/AppLogging.scala new file mode 100644 index 0000000..673d2f2 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/AppLogging.scala @@ -0,0 +1,8 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.actor.{ActorSystem, ExtendedActorSystem} +import akka.event.{BusLogging, LoggingAdapter} + +trait AppLogging { + def log(implicit system: ActorSystem): LoggingAdapter = new BusLogging(system.eventStream, this.getClass.getName, this.getClass, system.asInstanceOf[ExtendedActorSystem].logFilter) +} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala index 1906a77..a0792bf 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -1,6 +1,6 @@ package de.upb.cs.swt.delphi.webapi -import akka.actor.{Actor, ActorLogging, PoisonPill, Props, ReceiveTimeout} +import akka.actor.{Actor, ActorLogging, Props, ReceiveTimeout} import com.sksamuel.elastic4s.IndexAndType import com.sksamuel.elastic4s.http.ElasticDsl._ import com.sksamuel.elastic4s.http.HttpClient diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala index 5dbd562..b115d26 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala @@ -2,7 +2,7 @@ package de.upb.cs.swt.delphi.webapi import akka.actor.{Actor, ActorLogging, Props} import com.sksamuel.elastic4s.http.ElasticDsl._ -import de.upb.cs.swt.delphi.webapi.ElasticActorManager.Retrieve +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource class ElasticActorManager(configuration: Configuration) extends Actor with ActorLogging{ @@ -13,16 +13,21 @@ class ElasticActorManager(configuration: Configuration) extends Actor with Actor override def postStop(): Unit = log.info("Actor manager shut down") override def receive = { - case Retrieve(id) => { - log.info("Creating actor to search for entry {}", id) - val retrieveActor = context.actorOf(ElasticActor.props(configuration)) - retrieveActor forward GetSource(id, index) - } + case Retrieve(id) => getSource(id) + case Enqueue(id) => getSource(id) + } + + private def getSource(id: String) = { + log.info("Creating actor to search for entry {}", id) + val retrieveActor = context.actorOf(ElasticActor.props(configuration)) + retrieveActor forward GetSource(id, index) } } object ElasticActorManager{ def props(configuration: Configuration) : Props = Props(new ElasticActorManager(configuration)) + .withMailbox("es-priority-mailbox") final case class Retrieve(id: String) + final case class Enqueue(id: String) } \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticPriorityMailbox.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticPriorityMailbox.scala new file mode 100644 index 0000000..5600c01 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticPriorityMailbox.scala @@ -0,0 +1,14 @@ +package de.upb.cs.swt.delphi.webapi + +import akka.actor.ActorSystem +import akka.dispatch.{PriorityGenerator, UnboundedStablePriorityMailbox} +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} +import com.typesafe.config.Config + +class ElasticPriorityMailbox (settings: ActorSystem.Settings, config: Config) + extends UnboundedStablePriorityMailbox( + PriorityGenerator{ + case Retrieve(_) => 5 + case Enqueue(_) => 1 + case _ => 2 + }) diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index a736dd3..06b1f76 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -7,13 +7,13 @@ import akka.http.scaladsl.server.HttpApp import akka.pattern.ask import akka.util.Timeout import de.upb.cs.swt.delphi.featuredefinitions.FeatureListMapping -import de.upb.cs.swt.delphi.webapi.ElasticActorManager.Retrieve +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} import spray.json._ /** * Web server configuration for Delphi web API. */ -object Server extends HttpApp with JsonSupport { +object Server extends HttpApp with JsonSupport with AppLogging { private val configuration = new Configuration() private val system = ActorSystem("delphi-webapi") @@ -24,7 +24,8 @@ object Server extends HttpApp with JsonSupport { path("version") { version } ~ path("features") { features } ~ pathPrefix("search" / Remaining) { query => search(query) } ~ - pathPrefix("retrieve" / Remaining) { identifier => retrieve(identifier) } + pathPrefix("retrieve" / Remaining) { identifier => retrieve(identifier) } ~ + pathPrefix("enqueue" / Remaining) { identifier => enqueue(identifier) } private def version = { @@ -51,6 +52,16 @@ object Server extends HttpApp with JsonSupport { } } + def enqueue(identifier: String) = { + get { + pass { //TODO: Require authorization here + complete( + (actorManager ? Enqueue(identifier)).mapTo[String] + ) + } + } + } + def search(query: String) = { get { complete { From 39b03749cb51af3902214e1b093ebba73e22c1a8 Mon Sep 17 00:00:00 2001 From: almacken Date: Thu, 7 Jun 2018 14:46:31 -0600 Subject: [PATCH 13/34] Added Actor that limits request frequency by IP. This actor forwards requests to another actor, and blocks IPs for a certain amount of time if they make too many requests within a given window. It is not currently functioning, as there is not registration feature in this version, but it should work with a minor modification to the retrieve route. --- src/main/resources/application.conf | 8 ++ .../delphi/webapi/ElasticActorManager.scala | 4 +- .../cs/swt/delphi/webapi/ElasticMessage.scala | 3 + .../delphi/webapi/ElasticRequestLimiter.scala | 81 +++++++++++++++++++ .../de/upb/cs/swt/delphi/webapi/Server.scala | 14 +++- 5 files changed, 105 insertions(+), 5 deletions(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf index 9ebf7f5..4a97c1f 100644 --- a/src/main/resources/application.conf +++ b/src/main/resources/application.conf @@ -6,4 +6,12 @@ akka.actor.deployment { /espriomailboxactor { mailbox = es-priority-mailbox } +} + +akka { + http { + server { + remote-address-header = on + } + } } \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala index b115d26..756f2f3 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala @@ -28,6 +28,6 @@ object ElasticActorManager{ def props(configuration: Configuration) : Props = Props(new ElasticActorManager(configuration)) .withMailbox("es-priority-mailbox") - final case class Retrieve(id: String) - final case class Enqueue(id: String) + final case class Retrieve(id: String) extends ElasticMessage + final case class Enqueue(id: String) extends ElasticMessage } \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala new file mode 100644 index 0000000..f6aae33 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala @@ -0,0 +1,3 @@ +package de.upb.cs.swt.delphi.webapi + +trait ElasticMessage diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala new file mode 100644 index 0000000..df9998b --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala @@ -0,0 +1,81 @@ +package de.upb.cs.swt.delphi.webapi + + +import akka.actor.{Actor, ActorLogging, ActorRef, Props} +import akka.actor.Timers +import akka.http.scaladsl.model.RemoteAddress +import de.upb.cs.swt.delphi.webapi.ElasticRequestLimiter._ + +import scala.concurrent.duration._ +import scala.collection.mutable + +//Limits the number of requests any given IP can make by tracking how many requests an IP has made within a given +// window of time, and timing out any IP that exceeds a threshold by rejecting any further request for a period of time +class ElasticRequestLimiter(configuration: Configuration, nextActor: ActorRef) extends Actor with ActorLogging with Timers { + + private val window = 1 second + private val threshold = 10 + private val timeout = 2 hours + + private var recentIPs: mutable.Map[String, Int] = mutable.Map() + private var blockedIPs: mutable.Set[String] = mutable.Set() + + override def preStart(): Unit = { + log.info("Request limiter started") + timers.startPeriodicTimer(ClearTimer, ClearLogs, window) + } + override def postStop(): Unit = log.info("Request limiter shut down") + + override def receive = { + case Validate(rawIp, message) => { + val ip = rawIp.toOption.map(_.getHostAddress).getOrElse("unknown") + //First, reject IPs marked as blocked + if (blockedIPs.contains(ip)) { + rejectRequest() + } else { + //Check if this IP has made any requests recently + if (recentIPs.contains(ip)) { + //If so, increment their counter and test if they have exceeded the request threshold + recentIPs.update(ip, recentIPs(ip) + 1) + if (recentIPs(ip) > threshold) { + //If the threshold has been exceeded, mark this IP as blocked and reject it, and set up a message to unblock it after a period + blockedIPs += ip + log.info("Blocked IP {} due to exceeding request frequency threshold", ip) + timers.startSingleTimer(ForgiveTimer(ip), Forgive(ip), timeout) + rejectRequest() + } else { + //Else, forward this message + nextActor forward message + } + } else { + //Else, register their request in the map and pass it to the next actor + recentIPs += (ip -> 1) + nextActor forward message + } + } + } + case ClearLogs => + recentIPs.clear() + case Forgive(ip) => { + blockedIPs -= ip + log.info("Forgave IP {} after timeout", ip) + } + } + + //Rejects requests from blocked IPs + private def rejectRequest() = + sender() ! "Sorry, you have exceeded the limit on request frequency for unregistered users.\n" + + "As a result, you have been timed out.\n" + + "Please wait a while or register an account with us to continue using this service." +} + +object ElasticRequestLimiter{ + def props(configuration: Configuration, nextActor: ActorRef) : Props = Props(new ElasticRequestLimiter(configuration, nextActor)) + + final case class Validate(rawIp: RemoteAddress, message: ElasticMessage) + final case object ClearLogs + final case class Forgive(ip: String) + + final case object ClearTimer + final case class ForgiveTimer(ip: String) +} \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 06b1f76..02e72ee 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -8,6 +8,7 @@ import akka.pattern.ask import akka.util.Timeout import de.upb.cs.swt.delphi.featuredefinitions.FeatureListMapping import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} +import de.upb.cs.swt.delphi.webapi.ElasticRequestLimiter.Validate import spray.json._ /** @@ -18,6 +19,7 @@ object Server extends HttpApp with JsonSupport with AppLogging { private val configuration = new Configuration() private val system = ActorSystem("delphi-webapi") private val actorManager = system.actorOf(ElasticActorManager.props(configuration)) + private val requestLimiter = system.actorOf(ElasticRequestLimiter.props(configuration, actorManager)) implicit val timeout = Timeout(5, TimeUnit.SECONDS) override def routes = @@ -46,9 +48,15 @@ object Server extends HttpApp with JsonSupport with AppLogging { def retrieve(identifier: String) = { get { - complete( - (actorManager ? Retrieve(identifier)).mapTo[String] - ) + pass { //TODO: Require authentication here + complete( + (actorManager ? Retrieve(identifier)).mapTo[String] + ) + } ~ extractClientIP{ ip => + complete( + (requestLimiter ? Validate(ip, Retrieve(identifier))).mapTo[String] + ) + } } } From c6c53dc1f12358310c2efc292f719c564ef82e30 Mon Sep 17 00:00:00 2001 From: almacken Date: Tue, 12 Jun 2018 14:54:28 -0600 Subject: [PATCH 14/34] Improved actor routing --- src/main/resources/application.conf | 11 ++++++++ .../cs/swt/delphi/webapi/Configuration.scala | 13 ++++++--- .../cs/swt/delphi/webapi/ElasticActor.scala | 6 ++--- .../delphi/webapi/ElasticActorManager.scala | 27 ++++++++++++++----- .../cs/swt/delphi/webapi/ElasticMessage.scala | 3 --- .../delphi/webapi/ElasticRequestLimiter.scala | 1 + 6 files changed, 46 insertions(+), 15 deletions(-) delete mode 100644 src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf index 4a97c1f..935d3b3 100644 --- a/src/main/resources/application.conf +++ b/src/main/resources/application.conf @@ -14,4 +14,15 @@ akka { remote-address-header = on } } +} + +# Use this dispatcher for actors that make blocking calls to the Elasticsearch database +elasticsearch-handling-dispatcher { + type = Dispatcher + executor = "thread-pool-executor" + thread-pool-executor { + fixed-pool-size = 4 + # This thread pool is intended for development purposes, and should be increased for production + } + throughput = 1 } \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala index f61f6b6..53ed334 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Configuration.scala @@ -1,13 +1,20 @@ package de.upb.cs.swt.delphi.webapi -import com.sksamuel.elastic4s.ElasticsearchClientUri +import com.sksamuel.elastic4s.{ElasticsearchClientUri, IndexAndType} +import com.sksamuel.elastic4s.http.ElasticDsl._ /** * @author Ben Hermann */ -class Configuration(val bindHost: String = "0.0.0.0", +class Configuration( //Server and Elasticsearch configuration + val bindHost: String = "0.0.0.0", val bindPort: Int = 8080, val elasticsearchClientUri: ElasticsearchClientUri = ElasticsearchClientUri( - sys.env.getOrElse("DELPHI_ELASTIC_URI", "elasticsearch://localhost:9200"))) { + sys.env.getOrElse("DELPHI_ELASTIC_URI", "elasticsearch://localhost:9200")), + val esProjectIndex: IndexAndType = "delphi" / "project", + + //Actor system configuration + val elasticActorPoolSize: Int = 8 + ) { } diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala index a0792bf..d79df4c 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -1,15 +1,17 @@ package de.upb.cs.swt.delphi.webapi -import akka.actor.{Actor, ActorLogging, Props, ReceiveTimeout} +import akka.actor.{Actor, ActorLogging, Props} import com.sksamuel.elastic4s.IndexAndType import com.sksamuel.elastic4s.http.ElasticDsl._ import com.sksamuel.elastic4s.http.HttpClient import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource +import scala.concurrent.ExecutionContext import scala.concurrent.duration._ class ElasticActor(configuration: Configuration) extends Actor with ActorLogging{ + implicit val executionContext: ExecutionContext = context.system.dispatchers.lookup("elasticsearch-handling-dispatcher") val client = HttpClient(configuration.elasticsearchClientUri) override def preStart(): Unit = log.info("Search actor started") @@ -26,9 +28,7 @@ class ElasticActor(configuration: Configuration) extends Actor with ActorLogging case Left(_) => Option.empty } sender().tell(source, context.self) - context.stop(self) } - case ReceiveTimeout => context.stop(self) } } diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala index 756f2f3..77882b0 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala @@ -1,13 +1,21 @@ package de.upb.cs.swt.delphi.webapi -import akka.actor.{Actor, ActorLogging, Props} -import com.sksamuel.elastic4s.http.ElasticDsl._ +import akka.actor.{Actor, ActorLogging, Props, Terminated} +import akka.routing.{ActorRefRoutee, RoundRobinRoutingLogic, Router} import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource class ElasticActorManager(configuration: Configuration) extends Actor with ActorLogging{ - private val index = "delphi" / "project" + private val index = configuration.esProjectIndex + private var elasticRouter = { + val routees = Vector.fill(configuration.elasticActorPoolSize) { + val r = context.actorOf(ElasticActor.props(configuration)) + context watch r + ActorRefRoutee(r) + } + Router(RoundRobinRoutingLogic(), routees) + } override def preStart(): Unit = log.info("Actor manager started") override def postStop(): Unit = log.info("Actor manager shut down") @@ -15,12 +23,17 @@ class ElasticActorManager(configuration: Configuration) extends Actor with Actor override def receive = { case Retrieve(id) => getSource(id) case Enqueue(id) => getSource(id) + case Terminated(id) => { + elasticRouter.removeRoutee(id) + val r = context.actorOf(ElasticActor.props(configuration)) + context watch r + elasticRouter = elasticRouter.addRoutee(r) + } } private def getSource(id: String) = { - log.info("Creating actor to search for entry {}", id) - val retrieveActor = context.actorOf(ElasticActor.props(configuration)) - retrieveActor forward GetSource(id, index) + log.info("Forwarding search for entry {}", id) + elasticRouter.route(GetSource(id, index), sender()) } } @@ -28,6 +41,8 @@ object ElasticActorManager{ def props(configuration: Configuration) : Props = Props(new ElasticActorManager(configuration)) .withMailbox("es-priority-mailbox") + sealed trait ElasticMessage + final case class Retrieve(id: String) extends ElasticMessage final case class Enqueue(id: String) extends ElasticMessage } \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala deleted file mode 100644 index f6aae33..0000000 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticMessage.scala +++ /dev/null @@ -1,3 +0,0 @@ -package de.upb.cs.swt.delphi.webapi - -trait ElasticMessage diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala index df9998b..4395630 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticRequestLimiter.scala @@ -5,6 +5,7 @@ import akka.actor.{Actor, ActorLogging, ActorRef, Props} import akka.actor.Timers import akka.http.scaladsl.model.RemoteAddress import de.upb.cs.swt.delphi.webapi.ElasticRequestLimiter._ +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.ElasticMessage import scala.concurrent.duration._ import scala.collection.mutable From b933a800ccd8e4d09f09a84d53ee2928970952b1 Mon Sep 17 00:00:00 2001 From: almacken Date: Wed, 13 Jun 2018 14:41:21 -0600 Subject: [PATCH 15/34] Improved priority handling --- .../cs/swt/delphi/webapi/ElasticActor.scala | 30 ++++++++++--------- .../delphi/webapi/ElasticActorManager.scala | 18 +++++------ 2 files changed, 23 insertions(+), 25 deletions(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala index d79df4c..afe76fe 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -4,12 +4,12 @@ import akka.actor.{Actor, ActorLogging, Props} import com.sksamuel.elastic4s.IndexAndType import com.sksamuel.elastic4s.http.ElasticDsl._ import com.sksamuel.elastic4s.http.HttpClient -import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} import scala.concurrent.ExecutionContext import scala.concurrent.duration._ -class ElasticActor(configuration: Configuration) extends Actor with ActorLogging{ +class ElasticActor(configuration: Configuration, index: IndexAndType) extends Actor with ActorLogging{ implicit val executionContext: ExecutionContext = context.system.dispatchers.lookup("elasticsearch-handling-dispatcher") val client = HttpClient(configuration.elasticsearchClientUri) @@ -19,21 +19,23 @@ class ElasticActor(configuration: Configuration) extends Actor with ActorLogging context.setReceiveTimeout(2 seconds) override def receive = { - case GetSource(id, index) => { - log.info("Executing get on entry {}", id) - def source = client.execute{ - get(id).from(index) - }.await match { - case Right(res) => res.body.get - case Left(_) => Option.empty - } - sender().tell(source, context.self) + case Enqueue(id) => getSource(id) + case Retrieve(id) => getSource(id) + } + + private def getSource(id: String) = { + log.info("Executing get on entry {}", id) + def source = client.execute{ + get(id).from(index) + }.await match { + case Right(res) => res.body.get + case Left(_) => Option.empty } + sender().tell(source, context.self) } } object ElasticActor{ - def props(configuration: Configuration) : Props = Props(new ElasticActor(configuration)) - - final case class GetSource(id: String, index: IndexAndType) + def props(configuration: Configuration, index: IndexAndType) : Props = Props(new ElasticActor(configuration, index)) + .withMailbox("es-priority-mailbox") } diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala index 77882b0..aa8f6fc 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActorManager.scala @@ -2,15 +2,14 @@ package de.upb.cs.swt.delphi.webapi import akka.actor.{Actor, ActorLogging, Props, Terminated} import akka.routing.{ActorRefRoutee, RoundRobinRoutingLogic, Router} -import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} -import de.upb.cs.swt.delphi.webapi.ElasticActor.GetSource +import de.upb.cs.swt.delphi.webapi.ElasticActorManager.ElasticMessage class ElasticActorManager(configuration: Configuration) extends Actor with ActorLogging{ private val index = configuration.esProjectIndex private var elasticRouter = { val routees = Vector.fill(configuration.elasticActorPoolSize) { - val r = context.actorOf(ElasticActor.props(configuration)) + val r = context.actorOf(ElasticActor.props(configuration, index)) context watch r ActorRefRoutee(r) } @@ -21,20 +20,17 @@ class ElasticActorManager(configuration: Configuration) extends Actor with Actor override def postStop(): Unit = log.info("Actor manager shut down") override def receive = { - case Retrieve(id) => getSource(id) - case Enqueue(id) => getSource(id) + case em: ElasticMessage => { + log.info("Forwarding request {} to ElasticActor", em) + elasticRouter.route(em, sender()) + } case Terminated(id) => { elasticRouter.removeRoutee(id) - val r = context.actorOf(ElasticActor.props(configuration)) + val r = context.actorOf(ElasticActor.props(configuration, index)) context watch r elasticRouter = elasticRouter.addRoutee(r) } } - - private def getSource(id: String) = { - log.info("Forwarding search for entry {}", id) - elasticRouter.route(GetSource(id, index), sender()) - } } object ElasticActorManager{ From 4053613c6896bbc35dd344ae373a07d47e0b086f Mon Sep 17 00:00:00 2001 From: Hariharan Ramanathan Date: Tue, 3 Jul 2018 12:41:30 +0200 Subject: [PATCH 16/34] Adding scalastyle to delphi webapi --- build.sbt | 8 +++ project/plugins.sbt | 2 +- scalastyle-config.xml | 117 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 scalastyle-config.xml diff --git a/build.sbt b/build.sbt index 9293e5f..ded010a 100644 --- a/build.sbt +++ b/build.sbt @@ -29,6 +29,7 @@ libraryDependencies ++= Seq( lazy val webapi = (project in file(".")). enablePlugins(JavaAppPackaging). enablePlugins(DockerPlugin). + enablePlugins(ScalastylePlugin). settings ( dockerBaseImage := "openjdk:jre-alpine" ). @@ -38,3 +39,10 @@ lazy val webapi = (project in file(".")). buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, sbtVersion), buildInfoPackage := "de.upb.cs.swt.delphi.webapi" ) + +lazy val scalastyleTask = taskKey[Unit]("scalastyleTask") +scalastyleTask :={ + scalastyle.in(Compile).toTask("").value + scalastyle.in(Test).toTask("").value +} +(test in Test) := ((test in Test) dependsOn scalastyleTask).value diff --git a/project/plugins.sbt b/project/plugins.sbt index 2dcfb83..8d08441 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -1,2 +1,2 @@ addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.2") - +addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0") diff --git a/scalastyle-config.xml b/scalastyle-config.xml new file mode 100644 index 0000000..7e3596f --- /dev/null +++ b/scalastyle-config.xml @@ -0,0 +1,117 @@ + + Scalastyle standard configuration + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From 48a698042df1d2e76f901409f7448ff5ecfa5f26 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 26 Jul 2018 09:31:48 +0200 Subject: [PATCH 17/34] Finished license application --- LICENSE | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/LICENSE b/LICENSE index 261eeb9..d7e3293 100644 --- a/LICENSE +++ b/LICENSE @@ -175,18 +175,7 @@ END OF TERMS AND CONDITIONS - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] + Copyright 2018 The Delphi Team (represented by Ben Hermann) Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. From 9af235033f9940e8c4b49ca0e4e24f60bce0754b Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 26 Jul 2018 11:17:59 +0200 Subject: [PATCH 18/34] Added code coverage support and support for snyk --- .travis.yml | 7 ++++++- project/buildinfo.sbt | 1 - project/plugins.sbt | 8 ++++++++ 3 files changed, 14 insertions(+), 2 deletions(-) delete mode 100644 project/buildinfo.sbt diff --git a/.travis.yml b/.travis.yml index ab14e26..0a2dc6c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,3 +1,8 @@ language: scala scala: - - 2.12.4 \ No newline at end of file + - 2.12.4 +script: + - 'if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then sbt ++$TRAVIS_SCALA_VERSION test; fi' + - 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then sbt ++$TRAVIS_SCALA_VERSION coverage test coverageReport coverageAggregate codacyCoverage; fi' +after_success: + - 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash <(curl -s https://codecov.io/bash); fi' diff --git a/project/buildinfo.sbt b/project/buildinfo.sbt deleted file mode 100644 index 42c669b..0000000 --- a/project/buildinfo.sbt +++ /dev/null @@ -1 +0,0 @@ -addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.7.0") \ No newline at end of file diff --git a/project/plugins.sbt b/project/plugins.sbt index 2dcfb83..c3f46c6 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -1,2 +1,10 @@ +// build management and packaging +addSbtPlugin("com.eed3si9n" % "sbt-buildinfo" % "0.7.0") addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.2") +// coverage +addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.5.1") +addSbtPlugin("com.codacy" % "sbt-codacy-coverage" % "1.3.12") + +// preparation for dependency checking +addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.1") From 7236fc7dd15f4238d72e4328f1b3d84b822e6591 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 26 Jul 2018 13:27:35 +0200 Subject: [PATCH 19/34] Updated elasticsearch client in order to avoid vulnerable libraries in the dependecy tree --- build.sbt | 2 +- .../de/upb/cs/swt/delphi/webapi/ElasticActor.scala | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/build.sbt b/build.sbt index 9293e5f..db09d55 100644 --- a/build.sbt +++ b/build.sbt @@ -10,7 +10,7 @@ libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.12" libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.1.1" libraryDependencies += "io.spray" %% "spray-json" % "1.3.3" -val elastic4sVersion = "6.2.8" +val elastic4sVersion = "6.3.0" libraryDependencies ++= Seq( "com.sksamuel.elastic4s" %% "elastic4s-core" % elastic4sVersion, diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala index afe76fe..d0cdfd3 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -2,8 +2,8 @@ package de.upb.cs.swt.delphi.webapi import akka.actor.{Actor, ActorLogging, Props} import com.sksamuel.elastic4s.IndexAndType +import com.sksamuel.elastic4s.http.{ElasticClient, RequestFailure, RequestSuccess} import com.sksamuel.elastic4s.http.ElasticDsl._ -import com.sksamuel.elastic4s.http.HttpClient import de.upb.cs.swt.delphi.webapi.ElasticActorManager.{Enqueue, Retrieve} import scala.concurrent.ExecutionContext @@ -12,7 +12,7 @@ import scala.concurrent.duration._ class ElasticActor(configuration: Configuration, index: IndexAndType) extends Actor with ActorLogging{ implicit val executionContext: ExecutionContext = context.system.dispatchers.lookup("elasticsearch-handling-dispatcher") - val client = HttpClient(configuration.elasticsearchClientUri) + val client = ElasticClient(configuration.elasticsearchClientUri) override def preStart(): Unit = log.info("Search actor started") override def postStop(): Unit = log.info("Search actor shut down") @@ -25,11 +25,13 @@ class ElasticActor(configuration: Configuration, index: IndexAndType) extends Ac private def getSource(id: String) = { log.info("Executing get on entry {}", id) - def source = client.execute{ + def queryResponse = client.execute{ get(id).from(index) - }.await match { - case Right(res) => res.body.get - case Left(_) => Option.empty + }.await + + val source = queryResponse match { + case results: RequestSuccess[_] => results.body.get + case failure: RequestFailure => Option.empty } sender().tell(source, context.self) } From ec879b6c88de30864ba0b2a4bfe54678951ac059 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Thu, 26 Jul 2018 15:49:30 +0200 Subject: [PATCH 20/34] Activated scalastyle plugin --- build.sbt | 2 ++ project/plugins.sbt | 3 +++ 2 files changed, 5 insertions(+) diff --git a/build.sbt b/build.sbt index db09d55..a6655c0 100644 --- a/build.sbt +++ b/build.sbt @@ -38,3 +38,5 @@ lazy val webapi = (project in file(".")). buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, sbtVersion), buildInfoPackage := "de.upb.cs.swt.delphi.webapi" ) + +scalastyleConfig := baseDirectory.value / "project" / "scalastyle_config.xml" diff --git a/project/plugins.sbt b/project/plugins.sbt index c3f46c6..705f861 100644 --- a/project/plugins.sbt +++ b/project/plugins.sbt @@ -8,3 +8,6 @@ addSbtPlugin("com.codacy" % "sbt-codacy-coverage" % "1.3.12") // preparation for dependency checking addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.9.1") + +// scalastyle +addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0") From a6c40301abcdf688b930893fc9d0eea7592e6ce1 Mon Sep 17 00:00:00 2001 From: Hariharan Ramanathan Date: Tue, 31 Jul 2018 13:49:01 +0200 Subject: [PATCH 21/34] Moved scala style config to project folder --- build.sbt | 2 ++ scalastyle-config.xml => project/scalastyle-config.xml | 0 2 files changed, 2 insertions(+) rename scalastyle-config.xml => project/scalastyle-config.xml (100%) diff --git a/build.sbt b/build.sbt index ded010a..cea1247 100644 --- a/build.sbt +++ b/build.sbt @@ -45,4 +45,6 @@ scalastyleTask :={ scalastyle.in(Compile).toTask("").value scalastyle.in(Test).toTask("").value } +(scalastyleConfig in Compile):=file("project/scalastyle-config.xml") +(scalastyleConfig in Test):=file("project/scalastyle-config.xml") (test in Test) := ((test in Test) dependsOn scalastyleTask).value diff --git a/scalastyle-config.xml b/project/scalastyle-config.xml similarity index 100% rename from scalastyle-config.xml rename to project/scalastyle-config.xml From 273b85aa2c1ccc86a3f089f4774f0aa770a4af65 Mon Sep 17 00:00:00 2001 From: Lisa Nguyen Date: Tue, 31 Jul 2018 16:59:32 +0200 Subject: [PATCH 22/34] Added specification for the parser. Todo: Fix Test 2 that loops indefinitely, add more tests. --- build.sbt | 3 + .../delphi/querylanguage/ConditionExpr.scala | 20 +++++++ .../delphi/querylanguage/ConstantValue.scala | 10 ++++ .../cs/swt/delphi/querylanguage/Syntax.scala | 58 ++++++++++--------- .../swt/delphi/querylanguage/SyntaxTest.scala | 19 ++++++ 5 files changed, 82 insertions(+), 28 deletions(-) create mode 100644 src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala create mode 100644 src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala create mode 100644 src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala diff --git a/build.sbt b/build.sbt index a6655c0..2326634 100644 --- a/build.sbt +++ b/build.sbt @@ -9,6 +9,9 @@ libraryDependencies += "com.typesafe.akka" %% "akka-http" % "10.0.11" libraryDependencies += "com.typesafe.akka" %% "akka-stream" % "2.5.12" libraryDependencies += "com.typesafe.akka" %% "akka-http-spray-json" % "10.1.1" libraryDependencies += "io.spray" %% "spray-json" % "1.3.3" +libraryDependencies += "org.parboiled" %% "parboiled" % "2.1.4" +libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.4" +libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.4" % "test" val elastic4sVersion = "6.3.0" libraryDependencies ++= Seq( diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala new file mode 100644 index 0000000..dc643e3 --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala @@ -0,0 +1,20 @@ +package de.upb.cs.swt.delphi.querylanguage + +trait ConditionExpr extends Expression { + + def -->(expr: Expression): Expression = { + println("Evaluating: " + expr) + expr match { + case AndExpr(left, right) => AndExpr(left, right) + case OrExpr(left, right) => OrExpr(left, right) + case NotExpr(expr) => -->(expr) + case XorExpr(left, right) => XorExpr(left, right) + case _ => null + } + } +} + +case class AndExpr(Left: Expression, Right: Expression) extends ConditionExpr +case class OrExpr(Left: Expression, Right: Expression) extends ConditionExpr +case class NotExpr(Expr: Expression) extends ConditionExpr +case class XorExpr(Left: Expression, Right: Expression) extends ConditionExpr diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala new file mode 100644 index 0000000..ffcf74c --- /dev/null +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala @@ -0,0 +1,10 @@ +package de.upb.cs.swt.delphi.querylanguage + +trait Expression { +} + +class Constant(value: String) extends Expression { + override def toString: String = value +} + +case class ConstantValue(v : String) extends Constant(v) with ConditionExpr \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index d18e1be..923ab83 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -19,35 +19,34 @@ class Syntax(val input : ParserInput) extends Parser { capture(oneOrMore(CharPredicate.Alpha)) } - def ConditionRule = rule { - SingularConditionRule // | ConjunctionRule | DisjunctionRule + def ConditionRule : Rule1[ConditionExpr] = rule { + SingularConditionRule | AndRule | OrRule | NotRule | XorRule } def SingularConditionRule = rule { - EqualRule | - NotEqualRule | - GreaterThanRule | - GreaterOrEqual | - LessThan | - LessOrEqual | - Like + EqualRule | NotEqualRule | GreaterThanRule | GreaterOrEqual | LessThan | LessOrEqual | Like | True } - /* - def ConjunctionRule = rule { - ConditionRule ~ "&&" ~ ConditionRule - } + /* Combinatory rules */ - def DisjunctionRule = rule { - ConditionRule ~ "||" ~ ConditionRule + def AndRule = rule { + ConditionRule ~ "&&" ~ ConditionRule ~> AndExpr + } + def OrRule = rule { + ConditionRule ~ "||" ~ ConditionRule ~> OrExpr + } + def NotRule = rule { + "!" ~ ConditionRule ~> NotExpr + } + def XorRule = rule { + ConditionRule ~ "XX" ~ ConditionRule ~> XorExpr } - */ - def Literal = rule { - NumberLiteral | - StringLiteral + NumberLiteral | StringLiteral } + /* Literals */ + def StringLiteral = rule { capture(oneOrMore(CharPredicate.AlphaNum)) } @@ -56,28 +55,31 @@ class Syntax(val input : ParserInput) extends Parser { capture(oneOrMore(CharPredicate.Digit)) } + /* Value operators */ + def EqualRule = rule { - "=" ~ Literal + "=" ~ Literal ~> ConstantValue } def NotEqualRule = rule { - "!=" ~ NumberLiteral + "!=" ~ NumberLiteral ~> ConstantValue } def GreaterThanRule = rule { - ">" ~ NumberLiteral + ">" ~ NumberLiteral ~> ConstantValue } def GreaterOrEqual = rule { - ">=" ~ NumberLiteral + ">=" ~ NumberLiteral ~> ConstantValue } - def LessThan = rule { - "<" ~ NumberLiteral + "<" ~ NumberLiteral ~> ConstantValue } def LessOrEqual = rule { - "=<" ~ NumberLiteral + "<=" ~ NumberLiteral ~> ConstantValue } - def Like = rule { - "~" ~ StringLiteral + "%" ~ StringLiteral ~ "%" ~> ConstantValue + } + def True = rule { + StringLiteral ~> ConstantValue } } diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala new file mode 100644 index 0000000..4a18e6f --- /dev/null +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -0,0 +1,19 @@ +package de.upb.cs.swt.delphi.querylanguage + +import org.scalatest.{FlatSpec, Matchers} +import scala.util.{Failure, Success} + +class SyntaxTest extends FlatSpec with Matchers { + + "Syntax.oneFilterLessOrEqual" should "be valid" in { + val parseResult = new Syntax("Filter<=3").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + // TODO: Fix that. + "Syntax.oneFilterLessOrEqualTypo" should "be valid" in { + val parseResult = new Syntax("Filter=<3").QueryRule.run() + parseResult shouldBe a [Failure[_]] + } + +} \ No newline at end of file From f2c5b38a8148e9ccd04fa6dfea39a21e625c352f Mon Sep 17 00:00:00 2001 From: Lisa Nguyen Date: Wed, 1 Aug 2018 17:03:28 +0200 Subject: [PATCH 23/34] Fixed typo test case. Todo: Infinite loop still there. --- .../delphi/querylanguage/ConditionExpr.scala | 29 ++++++++++++++-- .../delphi/querylanguage/ConstantValue.scala | 10 ------ .../cs/swt/delphi/querylanguage/Syntax.scala | 33 +++++++------------ .../de/upb/cs/swt/delphi/webapi/Server.scala | 3 +- .../swt/delphi/querylanguage/SyntaxTest.scala | 24 +++++++++++--- 5 files changed, 61 insertions(+), 38 deletions(-) delete mode 100644 src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala index dc643e3..9437eda 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala @@ -1,9 +1,9 @@ package de.upb.cs.swt.delphi.querylanguage -trait ConditionExpr extends Expression { +trait Expression {} +trait ConditionExpr extends Expression { def -->(expr: Expression): Expression = { - println("Evaluating: " + expr) expr match { case AndExpr(left, right) => AndExpr(left, right) case OrExpr(left, right) => OrExpr(left, right) @@ -18,3 +18,28 @@ case class AndExpr(Left: Expression, Right: Expression) extends ConditionExpr case class OrExpr(Left: Expression, Right: Expression) extends ConditionExpr case class NotExpr(Expr: Expression) extends ConditionExpr case class XorExpr(Left: Expression, Right: Expression) extends ConditionExpr + +trait SingularConditionExpr extends ConditionExpr { + override def -->(expr: Expression): Expression = { + expr match { + case EqualExpr(left, right) => EqualExpr(left, right) + case NotEqualExpr(left, right) => NotEqualExpr(left, right) + case GreaterThanExpr(left, right) => GreaterThanExpr(left, right) + case GreaterOrEqualExpr(left, right) => GreaterOrEqualExpr(left, right) + case LessThanExpr(left, right) => LessThanExpr(left, right) + case LessOrEqualExpr(left, right) => LessOrEqualExpr(left, right) + case LikeExpr(left, right) => LikeExpr(left, right) + case TrueExpr(expr) => TrueExpr(expr) + case _ => null + } + } +} + +case class EqualExpr(Left: String, Right: String) extends SingularConditionExpr +case class NotEqualExpr(Left: String, Right: String) extends SingularConditionExpr +case class GreaterThanExpr(Left: String, Right: String) extends SingularConditionExpr +case class GreaterOrEqualExpr(Left: String, Right: String) extends SingularConditionExpr +case class LessThanExpr(Left: String, Right: String) extends SingularConditionExpr +case class LessOrEqualExpr(Left: String, Right: String) extends SingularConditionExpr +case class LikeExpr(Left: String, Right: String) extends SingularConditionExpr +case class TrueExpr(Expr: String) extends SingularConditionExpr diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala deleted file mode 100644 index ffcf74c..0000000 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConstantValue.scala +++ /dev/null @@ -1,10 +0,0 @@ -package de.upb.cs.swt.delphi.querylanguage - -trait Expression { -} - -class Constant(value: String) extends Expression { - override def toString: String = value -} - -case class ConstantValue(v : String) extends Constant(v) with ConditionExpr \ No newline at end of file diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index 923ab83..18fdaf4 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -8,15 +8,7 @@ import org.parboiled2.{CharPredicate, Parser, ParserInput, Rule1} class Syntax(val input : ParserInput) extends Parser { def QueryRule = rule { - QueryElementRule ~ EOI - } - - def QueryElementRule = rule { - FeatureIdentifierRule ~ ConditionRule - } - - def FeatureIdentifierRule = rule { - capture(oneOrMore(CharPredicate.Alpha)) + ConditionRule ~ EOI } def ConditionRule : Rule1[ConditionExpr] = rule { @@ -41,12 +33,13 @@ class Syntax(val input : ParserInput) extends Parser { def XorRule = rule { ConditionRule ~ "XX" ~ ConditionRule ~> XorExpr } + + /* Literals */ + def Literal = rule { NumberLiteral | StringLiteral } - /* Literals */ - def StringLiteral = rule { capture(oneOrMore(CharPredicate.AlphaNum)) } @@ -58,31 +51,29 @@ class Syntax(val input : ParserInput) extends Parser { /* Value operators */ def EqualRule = rule { - "=" ~ Literal ~> ConstantValue + StringLiteral ~ "=" ~ Literal ~> EqualExpr } def NotEqualRule = rule { - "!=" ~ NumberLiteral ~> ConstantValue + StringLiteral ~ "!=" ~ Literal ~> NotEqualExpr } def GreaterThanRule = rule { - ">" ~ NumberLiteral ~> ConstantValue + StringLiteral ~ ">" ~ NumberLiteral ~> GreaterThanExpr } def GreaterOrEqual = rule { - ">=" ~ NumberLiteral ~> ConstantValue + StringLiteral ~ ">=" ~ NumberLiteral ~> GreaterOrEqualExpr } def LessThan = rule { - "<" ~ NumberLiteral ~> ConstantValue + StringLiteral ~ "<" ~ NumberLiteral ~> LessThanExpr } def LessOrEqual = rule { - "<=" ~ NumberLiteral ~> ConstantValue + StringLiteral ~ "<=" ~ NumberLiteral ~> LessOrEqualExpr } def Like = rule { - "%" ~ StringLiteral ~ "%" ~> ConstantValue + StringLiteral ~ "%" ~ StringLiteral ~ "%" ~> LikeExpr } def True = rule { - StringLiteral ~> ConstantValue + StringLiteral ~> TrueExpr } } -class StringLiteral(value : String) {} -class NumberLiteral(value : Integer) {} diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index 02e72ee..c2aee8e 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -33,7 +33,8 @@ object Server extends HttpApp with JsonSupport with AppLogging { private def version = { get { complete { - BuildInfo.version + // BuildInfo.version + "Lala" } } } diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala index 4a18e6f..941542a 100644 --- a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -5,15 +5,31 @@ import scala.util.{Failure, Success} class SyntaxTest extends FlatSpec with Matchers { - "Syntax.oneFilterLessOrEqual" should "be valid" in { - val parseResult = new Syntax("Filter<=3").QueryRule.run() + "Syntax.oneFilterEqual" should "be valid" in { + val parseResult = new Syntax("Filter1=3").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + "Syntax.oneFilterTrue" should "be valid" in { + val parseResult = new Syntax("Filter1").QueryRule.run() parseResult shouldBe a [Success[_]] } - // TODO: Fix that. "Syntax.oneFilterLessOrEqualTypo" should "be valid" in { - val parseResult = new Syntax("Filter=<3").QueryRule.run() + val parseResult = new Syntax("Filter1=<3").QueryRule.run() parseResult shouldBe a [Failure[_]] } + // TODO: Fix infinite loop. + "Syntax.oneFilterFalse" should "be valid" in { + val parseResult = new Syntax("!Filter2").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + // TODO: Fix false test. Related to the infinite loop issue. + "Syntax.twoFiltersOr" should "be valid" in { + val parseResult = new Syntax("Filter1=3||Filter2=5").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + } \ No newline at end of file From 270146a10e70a18469d14badfcebf650b6974d67 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Wed, 1 Aug 2018 17:11:08 +0200 Subject: [PATCH 24/34] Rule precedence --- src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index 18fdaf4..2c5afe2 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -12,7 +12,7 @@ class Syntax(val input : ParserInput) extends Parser { } def ConditionRule : Rule1[ConditionExpr] = rule { - SingularConditionRule | AndRule | OrRule | NotRule | XorRule + SingularConditionRule | NotRule | AndRule | OrRule | XorRule } def SingularConditionRule = rule { From 7ae0ff048a543572d1fd45b8f62077b268313270 Mon Sep 17 00:00:00 2001 From: Lisa Nguyen Date: Wed, 1 Aug 2018 22:21:38 +0200 Subject: [PATCH 25/34] Basic parser finished and tested. Todo: add more tests. --- .../delphi/querylanguage/ConditionExpr.scala | 19 ++-- .../cs/swt/delphi/querylanguage/Syntax.scala | 90 +++++++------------ .../swt/delphi/querylanguage/SyntaxTest.scala | 53 ++++++++--- 3 files changed, 81 insertions(+), 81 deletions(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala index 9437eda..c69a181 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala @@ -1,9 +1,7 @@ package de.upb.cs.swt.delphi.querylanguage -trait Expression {} - -trait ConditionExpr extends Expression { - def -->(expr: Expression): Expression = { +trait CombinatorialExpr { + def -->(expr: CombinatorialExpr): CombinatorialExpr = { expr match { case AndExpr(left, right) => AndExpr(left, right) case OrExpr(left, right) => OrExpr(left, right) @@ -14,13 +12,13 @@ trait ConditionExpr extends Expression { } } -case class AndExpr(Left: Expression, Right: Expression) extends ConditionExpr -case class OrExpr(Left: Expression, Right: Expression) extends ConditionExpr -case class NotExpr(Expr: Expression) extends ConditionExpr -case class XorExpr(Left: Expression, Right: Expression) extends ConditionExpr +case class AndExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr +case class OrExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr +case class NotExpr(Expr: CombinatorialExpr) extends CombinatorialExpr +case class XorExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr -trait SingularConditionExpr extends ConditionExpr { - override def -->(expr: Expression): Expression = { +trait SingularConditionExpr extends CombinatorialExpr { + override def -->(expr: CombinatorialExpr): CombinatorialExpr = { expr match { case EqualExpr(left, right) => EqualExpr(left, right) case NotEqualExpr(left, right) => NotEqualExpr(left, right) @@ -29,7 +27,6 @@ trait SingularConditionExpr extends ConditionExpr { case LessThanExpr(left, right) => LessThanExpr(left, right) case LessOrEqualExpr(left, right) => LessOrEqualExpr(left, right) case LikeExpr(left, right) => LikeExpr(left, right) - case TrueExpr(expr) => TrueExpr(expr) case _ => null } } diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index 2c5afe2..854ae78 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -4,76 +4,46 @@ import org.parboiled2.{CharPredicate, Parser, ParserInput, Rule1} /** * Created by benhermann on 03.02.18. + * Author: Lisa Nguyen Quang Do 01.08.2018 */ class Syntax(val input : ParserInput) extends Parser { def QueryRule = rule { - ConditionRule ~ EOI + CombinatorialRule ~ EOI } - def ConditionRule : Rule1[ConditionExpr] = rule { - SingularConditionRule | NotRule | AndRule | OrRule | XorRule + // Combinatorial rules. + def CombinatorialRule : Rule1[CombinatorialExpr] = rule { + NotRule | + Factor ~ zeroOrMore( + "&&" ~ Factor ~> AndExpr | + "||" ~ Factor ~> OrExpr | + "%%" ~ Factor ~> XorExpr) } - def SingularConditionRule = rule { - EqualRule | NotEqualRule | GreaterThanRule | GreaterOrEqual | LessThan | LessOrEqual | Like | True - } - - /* Combinatory rules */ - - def AndRule = rule { - ConditionRule ~ "&&" ~ ConditionRule ~> AndExpr - } - def OrRule = rule { - ConditionRule ~ "||" ~ ConditionRule ~> OrExpr - } - def NotRule = rule { - "!" ~ ConditionRule ~> NotExpr - } - def XorRule = rule { - ConditionRule ~ "XX" ~ ConditionRule ~> XorExpr - } - - /* Literals */ - - def Literal = rule { - NumberLiteral | StringLiteral - } - - def StringLiteral = rule { - capture(oneOrMore(CharPredicate.AlphaNum)) + // Handling parentheses. + def Factor : Rule1[CombinatorialExpr] = rule { + Parentheses | SingularConditionRule | NotRule } + def Parentheses = rule { '(' ~ CombinatorialRule ~ ')' } + def NotRule = rule { '!' ~ (CombinatorialRule | Parentheses) ~> NotExpr } - def NumberLiteral = rule { - capture(oneOrMore(CharPredicate.Digit)) - } - - /* Value operators */ - - def EqualRule = rule { - StringLiteral ~ "=" ~ Literal ~> EqualExpr - } - def NotEqualRule = rule { - StringLiteral ~ "!=" ~ Literal ~> NotEqualExpr - } - def GreaterThanRule = rule { - StringLiteral ~ ">" ~ NumberLiteral ~> GreaterThanExpr - } - def GreaterOrEqual = rule { - StringLiteral ~ ">=" ~ NumberLiteral ~> GreaterOrEqualExpr - } - def LessThan = rule { - StringLiteral ~ "<" ~ NumberLiteral ~> LessThanExpr - } - def LessOrEqual = rule { - StringLiteral ~ "<=" ~ NumberLiteral ~> LessOrEqualExpr - } - def Like = rule { - StringLiteral ~ "%" ~ StringLiteral ~ "%" ~> LikeExpr - } - def True = rule { - StringLiteral ~> TrueExpr - } + // Singular conditions. + def SingularConditionRule = rule { + EqualRule | NotEqualRule | GreaterThanRule | GreaterOrEqual | + LessThan | LessOrEqual | Like | True + } + def EqualRule = rule { Literal ~ "=" ~ Literal ~> EqualExpr } + def NotEqualRule = rule { Literal ~ "!=" ~ Literal ~> NotEqualExpr } + def GreaterThanRule = rule { Literal ~ ">" ~ Literal ~> GreaterThanExpr } + def GreaterOrEqual = rule { Literal ~ ">=" ~ Literal ~> GreaterOrEqualExpr } + def LessThan = rule { Literal ~ "<" ~ Literal ~> LessThanExpr } + def LessOrEqual = rule { Literal ~ "<=" ~ Literal ~> LessOrEqualExpr } + def Like = rule { Literal ~ "%" ~ Literal ~> LikeExpr } + def True = rule { Literal ~> TrueExpr } + + // Literals + def Literal = rule { capture(oneOrMore(CharPredicate.AlphaNum)) ~> (_.toString) } } diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala index 941542a..5ae9134 100644 --- a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -5,30 +5,63 @@ import scala.util.{Failure, Success} class SyntaxTest extends FlatSpec with Matchers { - "Syntax.oneFilterEqual" should "be valid" in { - val parseResult = new Syntax("Filter1=3").QueryRule.run() + "Syntax.singularConditionWithOperator" should "be valid" in { + val parseResult = new Syntax("Filter1=abc").QueryRule.run() parseResult shouldBe a [Success[_]] } - "Syntax.oneFilterTrue" should "be valid" in { + "Syntax.singularConditionNoOperator" should "be valid" in { val parseResult = new Syntax("Filter1").QueryRule.run() parseResult shouldBe a [Success[_]] } - "Syntax.oneFilterLessOrEqualTypo" should "be valid" in { + "Syntax.singularConditionTypo" should "be valid" in { val parseResult = new Syntax("Filter1=<3").QueryRule.run() parseResult shouldBe a [Failure[_]] } - // TODO: Fix infinite loop. - "Syntax.oneFilterFalse" should "be valid" in { - val parseResult = new Syntax("!Filter2").QueryRule.run() + "Syntax.singularConditionOddCharacters" should "be valid" in { + val parseResult = new Syntax("Filter1%Filter2%'}.:").QueryRule.run() + parseResult shouldBe a [Failure[_]] + } + + "Syntax.combinatoryConditionSimple" should "be valid" in { + val parseResult = new Syntax("Filter1&&Filter2=3").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + "Syntax.combinatoryConditionParentheses" should "be valid" in { + val parseResult = new Syntax("Filter1||Filter2&&(Filter3<3||Filter4>0)").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + "Syntax.combinatoryConditionParenthesesComplex" should "be valid" in { + val parseResult = new Syntax("Filter1&&(Filter2<3||Filter2>0%%(Filter4&&Filter5))").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + "Syntax.combinatoryConditionNonMatchingParentheses" should "be valid" in { + val parseResult = new Syntax("Filter1&&(Filter2<3||Filter2>0%%(Filter4)").QueryRule.run() + parseResult shouldBe a [Failure[_]] + } + + "Syntax.combinatoryConditionTypo" should "be valid" in { + val parseResult = new Syntax("Filter1&Filter2<3)").QueryRule.run() + parseResult shouldBe a [Failure[_]] + } + + "Syntax.notConditionSimple" should "be valid" in { + val parseResult = new Syntax("Filter1&&!Filter2").QueryRule.run() + parseResult shouldBe a [Success[_]] + } + + "Syntax.notConditionSimpleParentheses" should "be valid" in { + val parseResult = new Syntax("!(Filter1&&Filter2)").QueryRule.run() parseResult shouldBe a [Success[_]] } - // TODO: Fix false test. Related to the infinite loop issue. - "Syntax.twoFiltersOr" should "be valid" in { - val parseResult = new Syntax("Filter1=3||Filter2=5").QueryRule.run() + "Syntax.notConditionComplex" should "be valid" in { + val parseResult = new Syntax("!!(Filter1)&&!(Filter2<=0||!(Filter3&&!Filter4%abc))").QueryRule.run() parseResult shouldBe a [Success[_]] } From de8b79f4ed1e8d11be5b3d959135ebe66ddf6c1c Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Fri, 3 Aug 2018 14:54:41 +0200 Subject: [PATCH 26/34] Slimmed down and harmonized --- .../{ConditionExpr.scala => AST.scala} | 27 ++----------------- .../cs/swt/delphi/querylanguage/Syntax.scala | 7 +++-- .../swt/delphi/querylanguage/SyntaxTest.scala | 5 ++++ 3 files changed, 12 insertions(+), 27 deletions(-) rename src/main/scala/de/upb/cs/swt/delphi/querylanguage/{ConditionExpr.scala => AST.scala} (51%) diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/AST.scala similarity index 51% rename from src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala rename to src/main/scala/de/upb/cs/swt/delphi/querylanguage/AST.scala index c69a181..57b8abe 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/ConditionExpr.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/AST.scala @@ -1,36 +1,13 @@ package de.upb.cs.swt.delphi.querylanguage -trait CombinatorialExpr { - def -->(expr: CombinatorialExpr): CombinatorialExpr = { - expr match { - case AndExpr(left, right) => AndExpr(left, right) - case OrExpr(left, right) => OrExpr(left, right) - case NotExpr(expr) => -->(expr) - case XorExpr(left, right) => XorExpr(left, right) - case _ => null - } - } -} +trait CombinatorialExpr case class AndExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr case class OrExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr case class NotExpr(Expr: CombinatorialExpr) extends CombinatorialExpr case class XorExpr(Left: CombinatorialExpr, Right: CombinatorialExpr) extends CombinatorialExpr -trait SingularConditionExpr extends CombinatorialExpr { - override def -->(expr: CombinatorialExpr): CombinatorialExpr = { - expr match { - case EqualExpr(left, right) => EqualExpr(left, right) - case NotEqualExpr(left, right) => NotEqualExpr(left, right) - case GreaterThanExpr(left, right) => GreaterThanExpr(left, right) - case GreaterOrEqualExpr(left, right) => GreaterOrEqualExpr(left, right) - case LessThanExpr(left, right) => LessThanExpr(left, right) - case LessOrEqualExpr(left, right) => LessOrEqualExpr(left, right) - case LikeExpr(left, right) => LikeExpr(left, right) - case _ => null - } - } -} +trait SingularConditionExpr extends CombinatorialExpr case class EqualExpr(Left: String, Right: String) extends SingularConditionExpr case class NotEqualExpr(Left: String, Right: String) extends SingularConditionExpr diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index 854ae78..d486d6a 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -3,8 +3,11 @@ package de.upb.cs.swt.delphi.querylanguage import org.parboiled2.{CharPredicate, Parser, ParserInput, Rule1} /** - * Created by benhermann on 03.02.18. - * Author: Lisa Nguyen Quang Do 01.08.2018 + * The syntax definition and parser for the Delphi QL. + * + * @author Lisa Nguyen Quang Do + * @author Ben Hermann + * */ class Syntax(val input : ParserInput) extends Parser { diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala index 5ae9134..fc8dd83 100644 --- a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -3,6 +3,11 @@ package de.upb.cs.swt.delphi.querylanguage import org.scalatest.{FlatSpec, Matchers} import scala.util.{Failure, Success} +/** + * Tests for the DelphiQL syntax. + * + * @author Lisa Nguyen Quang Do + */ class SyntaxTest extends FlatSpec with Matchers { "Syntax.singularConditionWithOperator" should "be valid" in { From 1b428164b4fa61ce3a74eb90892bcb0f6295b59d Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Fri, 3 Aug 2018 14:59:00 +0200 Subject: [PATCH 27/34] Reactivated version information... tz tz tz. --- src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala index c2aee8e..02e72ee 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/Server.scala @@ -33,8 +33,7 @@ object Server extends HttpApp with JsonSupport with AppLogging { private def version = { get { complete { - // BuildInfo.version - "Lala" + BuildInfo.version } } } From 2d9561cf037959b6e394b6194a07295a89a0fb67 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Fri, 3 Aug 2018 15:24:18 +0200 Subject: [PATCH 28/34] Implemented retrieve command --- .../scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala index d0cdfd3..c4d4dcd 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/webapi/ElasticActor.scala @@ -26,7 +26,10 @@ class ElasticActor(configuration: Configuration, index: IndexAndType) extends Ac private def getSource(id: String) = { log.info("Executing get on entry {}", id) def queryResponse = client.execute{ - get(id).from(index) + log.info(s"Got retrieve request for $id.") + searchWithType(index) query must ( + matchQuery("name", s"http://repo1.maven.org/maven2/:$id") + ) }.await val source = queryResponse match { From 06f136dc0ea8551fb93966013e8fcaf4eb0c9426 Mon Sep 17 00:00:00 2001 From: Lisa Nguyen Date: Fri, 3 Aug 2018 23:32:17 +0200 Subject: [PATCH 29/34] Added more tests to the Delphi QL, fixed the operator priority. Now handles multiple '!' --- .../cs/swt/delphi/querylanguage/Syntax.scala | 19 +++-- .../swt/delphi/querylanguage/SyntaxTest.scala | 83 ++++++++++++++++++- 2 files changed, 93 insertions(+), 9 deletions(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index d486d6a..c9f76bf 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -17,11 +17,16 @@ class Syntax(val input : ParserInput) extends Parser { // Combinatorial rules. def CombinatorialRule : Rule1[CombinatorialExpr] = rule { - NotRule | - Factor ~ zeroOrMore( - "&&" ~ Factor ~> AndExpr | - "||" ~ Factor ~> OrExpr | - "%%" ~ Factor ~> XorExpr) + OrOrElseRule | NotRule + } + def OrOrElseRule = rule { + AndOrElseRule ~ zeroOrMore("||" ~ AndOrElseRule ~> OrExpr) + } + def AndOrElseRule = rule { + XorOrElseRule ~ zeroOrMore("&&" ~ XorOrElseRule ~> AndExpr) + } + def XorOrElseRule = rule { + Factor ~ zeroOrMore("%%" ~ Factor ~> XorExpr) } // Handling parentheses. @@ -29,7 +34,9 @@ class Syntax(val input : ParserInput) extends Parser { Parentheses | SingularConditionRule | NotRule } def Parentheses = rule { '(' ~ CombinatorialRule ~ ')' } - def NotRule = rule { '!' ~ (CombinatorialRule | Parentheses) ~> NotExpr } + def NotRule : Rule1[CombinatorialExpr] = rule { + '!' ~ (NotRule | SingularConditionRule | Parentheses) ~> NotExpr + } // Singular conditions. def SingularConditionRule = rule { diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala index fc8dd83..3b2bb75 100644 --- a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -13,11 +13,21 @@ class SyntaxTest extends FlatSpec with Matchers { "Syntax.singularConditionWithOperator" should "be valid" in { val parseResult = new Syntax("Filter1=abc").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "EqualExpr(Filter1,abc)" + } + } } "Syntax.singularConditionNoOperator" should "be valid" in { val parseResult = new Syntax("Filter1").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "TrueExpr(Filter1)" + } + } } "Syntax.singularConditionTypo" should "be valid" in { @@ -33,16 +43,33 @@ class SyntaxTest extends FlatSpec with Matchers { "Syntax.combinatoryConditionSimple" should "be valid" in { val parseResult = new Syntax("Filter1&&Filter2=3").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(TrueExpr(Filter1),EqualExpr(Filter2,3))" + } + } } "Syntax.combinatoryConditionParentheses" should "be valid" in { - val parseResult = new Syntax("Filter1||Filter2&&(Filter3<3||Filter4>0)").QueryRule.run() + val parseResult = new Syntax("Filter1||(Filter2&&(Filter3<3||Filter4>0))").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "OrExpr(TrueExpr(Filter1),AndExpr(TrueExpr(Filter2)," + + "OrExpr(LessThanExpr(Filter3,3),GreaterThanExpr(Filter4,0))))" + } + } } "Syntax.combinatoryConditionParenthesesComplex" should "be valid" in { - val parseResult = new Syntax("Filter1&&(Filter2<3||Filter2>0%%(Filter4&&Filter5))").QueryRule.run() + val parseResult = new Syntax("Filter1&&((Filter2<3||Filter2>0)%%(Filter4&&Filter5))").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(TrueExpr(Filter1),XorExpr(OrExpr(LessThanExpr(Filter2,3)," + + "GreaterThanExpr(Filter2,0)),AndExpr(TrueExpr(Filter4),TrueExpr(Filter5))))" + } + } } "Syntax.combinatoryConditionNonMatchingParentheses" should "be valid" in { @@ -55,19 +82,69 @@ class SyntaxTest extends FlatSpec with Matchers { parseResult shouldBe a [Failure[_]] } + "Syntax.combinatoryConditionLeftToRightPriority" should "be valid" in { + val parseResult = new Syntax("Filter1&&Filter2&&Filter3").QueryRule.run() + parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(AndExpr(TrueExpr(Filter1)," + + "TrueExpr(Filter2)),TrueExpr(Filter3))" + } + } + } + + "Syntax.combinatoryConditionOperatorPriorities" should "be valid" in { + val parseResult = new Syntax("Filter1||Filter2%%!Filter3&&Filter4").QueryRule.run() + parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "OrExpr(TrueExpr(Filter1),AndExpr(XorExpr(" + + "TrueExpr(Filter2),NotExpr(TrueExpr(Filter3))),TrueExpr(Filter4)))" + } + } + } + + "Syntax.combinatoryConditionOperatorPrioritiesParentheses" should "be valid" in { + val parseResult = new Syntax("(Filter1||Filter2)&&!Filter3%%!(Filter4&&Filter5)").QueryRule.run() + parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(OrExpr(TrueExpr(Filter1),TrueExpr(Filter2))," + + "XorExpr(NotExpr(TrueExpr(Filter3)),NotExpr(AndExpr(TrueExpr(Filter4)," + + "TrueExpr(Filter5)))))" + } + } + } + "Syntax.notConditionSimple" should "be valid" in { val parseResult = new Syntax("Filter1&&!Filter2").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(TrueExpr(Filter1),NotExpr(TrueExpr(Filter2)))" + } + } } "Syntax.notConditionSimpleParentheses" should "be valid" in { val parseResult = new Syntax("!(Filter1&&Filter2)").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "NotExpr(AndExpr(TrueExpr(Filter1),TrueExpr(Filter2)))" + } + } } "Syntax.notConditionComplex" should "be valid" in { val parseResult = new Syntax("!!(Filter1)&&!(Filter2<=0||!(Filter3&&!Filter4%abc))").QueryRule.run() parseResult shouldBe a [Success[_]] + parseResult match { + case Success(ast) => { + ast.toString shouldEqual "AndExpr(NotExpr(NotExpr(TrueExpr(Filter1)))," + + "NotExpr(OrExpr(LessOrEqualExpr(Filter2,0),NotExpr(AndExpr(TrueExpr(Filter3)," + + "NotExpr(LikeExpr(Filter4,abc)))))))" + } + } } - } \ No newline at end of file From e194b712a66ca1146cf2bc201e59379f76cb6883 Mon Sep 17 00:00:00 2001 From: Lisa Nguyen Date: Sat, 4 Aug 2018 07:48:12 +0200 Subject: [PATCH 30/34] Beautification, better tests --- .../de/upb/cs/swt/delphi/querylanguage/Syntax.scala | 4 +--- .../upb/cs/swt/delphi/querylanguage/SyntaxTest.scala | 10 ++++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala index c9f76bf..1779b56 100644 --- a/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala +++ b/src/main/scala/de/upb/cs/swt/delphi/querylanguage/Syntax.scala @@ -34,9 +34,7 @@ class Syntax(val input : ParserInput) extends Parser { Parentheses | SingularConditionRule | NotRule } def Parentheses = rule { '(' ~ CombinatorialRule ~ ')' } - def NotRule : Rule1[CombinatorialExpr] = rule { - '!' ~ (NotRule | SingularConditionRule | Parentheses) ~> NotExpr - } + def NotRule = rule { '!' ~ Factor ~> NotExpr } // Singular conditions. def SingularConditionRule = rule { diff --git a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala index 3b2bb75..336d420 100644 --- a/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala +++ b/src/test/scala/de/upb/cs/swt/delphi/querylanguage/SyntaxTest.scala @@ -117,21 +117,23 @@ class SyntaxTest extends FlatSpec with Matchers { } "Syntax.notConditionSimple" should "be valid" in { - val parseResult = new Syntax("Filter1&&!Filter2").QueryRule.run() + val parseResult = new Syntax("!Filter1&&!(Filter2)").QueryRule.run() parseResult shouldBe a [Success[_]] parseResult match { case Success(ast) => { - ast.toString shouldEqual "AndExpr(TrueExpr(Filter1),NotExpr(TrueExpr(Filter2)))" + ast.toString shouldEqual "AndExpr(NotExpr(TrueExpr(Filter1))," + + "NotExpr(TrueExpr(Filter2)))" } } } "Syntax.notConditionSimpleParentheses" should "be valid" in { - val parseResult = new Syntax("!(Filter1&&Filter2)").QueryRule.run() + val parseResult = new Syntax("!(Filter1&&!Filter2)").QueryRule.run() parseResult shouldBe a [Success[_]] parseResult match { case Success(ast) => { - ast.toString shouldEqual "NotExpr(AndExpr(TrueExpr(Filter1),TrueExpr(Filter2)))" + ast.toString shouldEqual "NotExpr(AndExpr(TrueExpr(Filter1)," + + "NotExpr(TrueExpr(Filter2))))" } } } From 993e4a55f59aff6141050d6fe5f3c5fe5f4f90f2 Mon Sep 17 00:00:00 2001 From: Hariharan Ramanathan Date: Thu, 9 Aug 2018 14:40:40 +0200 Subject: [PATCH 31/34] Removing scalastyle from test target --- build.sbt | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/build.sbt b/build.sbt index cea1247..20beb15 100644 --- a/build.sbt +++ b/build.sbt @@ -39,12 +39,6 @@ lazy val webapi = (project in file(".")). buildInfoKeys := Seq[BuildInfoKey](name, version, scalaVersion, sbtVersion), buildInfoPackage := "de.upb.cs.swt.delphi.webapi" ) +scalastyleConfig := baseDirectory.value / "project" / "scalastyle-config.xml" + -lazy val scalastyleTask = taskKey[Unit]("scalastyleTask") -scalastyleTask :={ - scalastyle.in(Compile).toTask("").value - scalastyle.in(Test).toTask("").value -} -(scalastyleConfig in Compile):=file("project/scalastyle-config.xml") -(scalastyleConfig in Test):=file("project/scalastyle-config.xml") -(test in Test) := ((test in Test) dependsOn scalastyleTask).value From c6fb25bf0dae0be078644f6d9a80d87660aa5600 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Sat, 11 Aug 2018 16:05:40 +0200 Subject: [PATCH 32/34] Update build.sbt --- build.sbt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.sbt b/build.sbt index a43415b..c9ab5ec 100644 --- a/build.sbt +++ b/build.sbt @@ -43,4 +43,4 @@ lazy val webapi = (project in file(".")). buildInfoPackage := "de.upb.cs.swt.delphi.webapi" ) -scalastyleConfig := baseDirectory.value / "project" / "scalastyle_config.xml" +scalastyleConfig := baseDirectory.value / "project" / "scalastyle-config.xml" From 576c98848855d5989c159eaa9a6a1817237cf032 Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Sat, 11 Aug 2018 16:05:55 +0200 Subject: [PATCH 33/34] Rename scalastyle_config.xml to scalastyle-config.xml --- project/{scalastyle_config.xml => scalastyle-config.xml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename project/{scalastyle_config.xml => scalastyle-config.xml} (100%) diff --git a/project/scalastyle_config.xml b/project/scalastyle-config.xml similarity index 100% rename from project/scalastyle_config.xml rename to project/scalastyle-config.xml From d876af9ebc67df7e9f79684b2ef70d1ea233553c Mon Sep 17 00:00:00 2001 From: Ben Hermann Date: Sat, 11 Aug 2018 16:17:43 +0200 Subject: [PATCH 34/34] Added README --- README.md | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..4c912c8 --- /dev/null +++ b/README.md @@ -0,0 +1,57 @@ +# Delphi Web API + +The web API implementation for the Delphi platform. + +We are currently in pre-alpha state! There is no release and the code in +this repository is purely experimental! + +|branch | status | codacy | +| :---: | :---: | :---: | +| master | [![Build Status](https://travis-ci.org/delphi-hub/delphi-webapi.svg?branch=master)](https://travis-ci.org/delphi-hub/delphi-webapi) | [![Codacy Badge](https://api.codacy.com/project/badge/Grade/8ebe27850ffb4139af6280fd1cd6d540)](https://www.codacy.com/app/delphi-hub/delphi-webapi?utm_source=github.com&utm_medium=referral&utm_content=delphi-hub/delphi-webapi&utm_campaign=Badge_Grade)| +| develop | [![Build Status](https://travis-ci.org/delphi-hub/delphi-webapi.svg?branch=develop)](https://travis-ci.org/delphi-hub/delphi-webapi) | [![Codacy Badge](https://api.codacy.com/project/badge/Grade/8ebe27850ffb4139af6280fd1cd6d540?branch=develop)](https://www.codacy.com/app/delphi-hub/delphi-webapi?branch=develop&utm_source=github.com&utm_medium=referral&utm_content=delphi-hub/delphi-webapi&utm_campaign=Badge_Grade) | + +## What is the Delphi Web API? + +It is the primary access point to the Delphi system. + +## How does it work? + +The Delphi Web API communicates with the underlying Elasticsearch database to provide access to data. + +## How can I use it? + +If you just wish to query the results, maybe the public instance at https://delphi.cs.uni-paderborn.de is the right choice for you. + +If you want to run your own infrastructure, you can start the web API +``` +sbt run +``` + +It expects a running instance of elasticsearch on port 9200 on the same machine. + +## Community + +Feel welcome to join our chatroom on Gitter: [![Join the chat at https://gitter.im/delphi-hub/delphi](https://badges.gitter.im/delphi-hub/delphi.svg)](https://gitter.im/delphi-hub/delphi?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) + + +## Contributing + +Contributions are *very* welcome! + +Before contributing, please read our [Code of Conduct](CODE_OF_CONDUCT.md). + +Refer to the [Contribution Guide](CONTRIBUTING.md) for details about the workflow. +We use Pull Requests to collect contributions. Especially look out for "help wanted" issues +[![GitHub issues by-label](https://img.shields.io/github/issues/delphi-hub/delphi-webapi/help%20wanted.svg)](https://github.com/delphi-hub/delphi-webapi/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22), +but feel free to work on other issues as well. +You can ask for clarification in the issues directly, or use our Gitter +chat for a more interactive experience. + +[![GitHub issues](https://img.shields.io/github/issues/delphi-hub/delphi-webapi.svg)](https://github.com/delphi-hub/delphi-webapi/issues) + + +## License + +The Delphi Web API is open source and available under Apache 2 License. + +[![GitHub license](https://img.shields.io/github/license/delphi-hub/delphi-webapi.svg)](https://github.com/delphi-hub/delphi-webapi/blob/master/LICENSE)