Asiainfo Spark_dev

Configuration

conf/spark-env.sh

  export HADOOP_CONF_DIR=/home/ocdc/hadoop-2.3.0-cdh5.0.0-och3.1.0/etc/hadoop
  export SPARK_YARN_APP_JAR=/home/ocdc/spark_0.9.1_streaming/examples/target/scala-2.10/spark-examples-assembly-0.9.1.jar
  export SPARK_JAR=/home/ocdc/spark_0.9.1_streaming/assembly/target/scala-2.10/spark-assembly-0.9.1-hadoop2.3.0-cdh5.0.0.jar
  

conf/Sample.xml

  ................
  
     
        com.asiainfo.ocdc.streaming.impl.KafkaSource
        
        Specifies the ZooKeeper connection string in the form hostname:port,
        where hostname and port are the host and port for a node in your ZooKeeper cluster.
        To allow connecting through other ZooKeeper nodes when that host is down
        you can also specify multiple hosts in the form 
        hostname1:port1,hostname2:port2,hostname3:port3
        cmbb3
        topicProducername
        test-consumer-group
        3
         
        a,b,c,d,e,f,count,fee
    

    
        com.asiainfo.ocdc.streaming.impl.StreamFilter
        t1
        cell
        lac,cell
        b,c,t1.cell,count,fee
        t1.cell!=null
    

    
        com.asiainfo.ocdc.streaming.impl.DynamicOperate
        t2
        b
        Count,Fee
        t2.Count+count,t2.Fee+fee
        b,c,t1.cell,count,fee
    

    
        com.asiainfo.ocdc.streaming.impl.KafkaOut
        topicName
        topic ConsumerName
        dev001:9092
        The port the socket server listens on,
        hostname1:port1,hostname2:port2,hostname3:port3
        b,c
    
    ......
  

Building Spark

  mvn package
  

Start spark streaming

The command format is as follows,To start the streaming app application

./bin/start-streaming-app.sh streaming-app-name 2 conf/Sample.xml

Parameter 1, execute the script file Parameter 2, When should the different configuration XML file, the corresponding streamingappname Parameter 3, flow interval refresh time(seconds) Parameter 4, The configuration file

Running Tests

Test documentation, reference spark_dev project wiki