Gtjkyu

Spark's Column.isin function does not take List I am trying to filter out rows from my Spark Dataframe. val sequence = Seq(1,2,3,4,5) df.filter(df("column").isin(sequence)) Unfortunately, I get an unsupported literal type error java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(1,2,3,4,5) according to the documentation it takes a scala.collection.Seq list I guess I don't want a literal? Then what can I take in, some sort of wrapper class? 2 Answers 2 @JustinPihony's answer is correct but it's incomplete. The isin function takes a repeated parameter for argument, so you'll need to pass it as so : isin scala> val df = sc.parallelize(Seq(1,2,3,4,5,6,7,8,9)).toDF("column") // df: org.apache.spark.sql.DataFrame = [column: int] scala> val sequence = Seq(1,2,3,4,5) // sequence: Seq[Int] = List(1, 2, 3, 4, 5...

Scala deserialize JSON to Collection My JSON File containes below details { "category":"age, gender,post_code" } My scala code is below one val filename = args.head println(s"Reading ${args.head} ...") val json = Source.fromFile(filename) val mapper = new ObjectMapper() with ScalaObjectMapper mapper.registerModule(DefaultScalaModule) val parsedJson = mapper.readValue[Map[String, Any]](json.reader()) val data = parsedJson.get("category").toSeq It's returning Seq(Any) = example List(age, gender,post_code) but I need Seq(String) output please if any has an idea about this please help me. 2 Answers 2 The idea in scala is to be typesafe whenever possible which you are giving away using Map[String, Any] . Map[String, Any] So, I recommend using a data class that represents your JSON data. Example, define a mapper, scala> import com.fasterxml.jackson.databind...

Scala - Perform Operation on two objects only if an inner field is equal If I have Class Box[+T] { num: Int, t: T } And I want to make a method which adds two boxes together, but really it just adds num and creates a new Box, how would I do that but ensuring both t's are equal? I don't want just the type to be the same, but the inner part of t to be the same 1 Answer 1 you can do the following: def add[T](box1: Box[T], box2: Box[T]): Option[Box[T]] = { if(box1.t == box2.t) Some(Box(box1.num + box2.num, box1.t)) else None } does this work in the context of this stackoverflow.com/questions/12700693/compare-types-in-scala – Joel Berkeley Jul 1 at 18:43 @JoelBerkeley I think it should. Also, this is OK I suppose....

Getting Bootstrap broker ip:9092 disconnected error from kafka spout Versions: "org.apache.storm" % "storm-kafka-client" % "1.2.1" "org.apache.storm" % "storm-core" % "1.2.1" % "compile" Kafka: 0.10.1.0 I am getting following error/warnings, running in localCluster, from my kafka spout: 2018-06-28 00:00:34,930 AppInfoParser [INFO] Kafka version : 0.10.1.0 2018-06-28 00:00:34,930 AppInfoParser [INFO] Kafka commitId : 3402a74efb23d1d4 2018-06-28 00:00:34,931 WARN NetworkClient [Thread-40-KafkaSpout-executor[12 12]] Bootstrap broker ip1:9092 disconnected 2018-06-28 00:00:35,092 WARN NetworkClient [Thread-40-KafkaSpout-executor[12 12]] Bootstrap broker ip2:9092 disconnected 2018-06-28 00:00:35,251 WARN NetworkClient [Thread-40-KafkaSpout-executor[12 12]] Bootstrap broker ip3:9092 disconnected 2018-06-28 00:00:35,524 WARN NetworkClient [Thread-40-KafkaSpout-executor[12 12]] Bootstrap broker ip4:9092 disconnected 2018-06-2...

Trouble with deserializing Avro data in Scala I am building an Apache Flink application in Scala which reads streaming data from a Kafka bus and then performs summarizing operations on it. The data from Kafka is in Avro format and needs a special Deserialization class. I found this scala class AvroDeserializationScehema (http://codegists.com/snippet/scala/avrodeserializationschemascala_saveveltri_scala): package org.myorg.quickstart import org.apache.avro.io.BinaryDecoder import org.apache.avro.io.DatumReader import org.apache.avro.io.DecoderFactory import org.apache.avro.reflect.ReflectDatumReader import org.apache.avro.specific.{SpecificDatumReader, SpecificRecordBase} import org.apache.flink.api.common.typeinfo.TypeInformation import org.apache.flink.api.java.typeutils.TypeExtractor import org.apache.flink.api.common.serialization._ import java.io.IOException class AvroDeserializationSchema[T](val avroType: Class[T]) extends DeserializationSchema[T] { private var reader: DatumRead...

Can I zip more than two lists together in Scala? Given the following Scala List: val l = List(List("a1", "b1", "c1"), List("a2", "b2", "c2"), List("a3", "b3", "c3")) How can I get: List(("a1", "a2", "a3"), ("b1", "b2", "b3"), ("c1", "c2", "c3")) Since zip can only be used to combine two Lists, I think you would need to iterate/reduce the main List somehow. Not surprisingly, the following doesn't work: scala> l reduceLeft ((a, b) => a zip b) <console>:6: error: type mismatch; found : List[(String, String)] required: List[String] l reduceLeft ((a, b) => a zip b) Any suggestions one how to do this? I think I'm missing a very simple way to do it. Update: I'm looking for a solution that can take a List of N Lists with M elements each and create a List of M TupleNs. Update 2: ...

(Scala) Am I using Options correctly? I'm currently working on my functional programming - I am fairly new to it. Am i using Options correctly here? I feel pretty insecure on my skills currently. I want my code to be as safe as possible - Can any one point out what am I doing wrong here or is it not that bad? My code is pretty straight forward here: def main(args: Array[String]): Unit = { val file = "myFile.txt" val myGame = Game(file) //I have my game that returns an Option here if(myGame.isDefined) //Check if I indeed past a .txt file { val solutions = myGame.get.getAllSolutions() //This returns options as well if(solutions.isDefined) //Is it possible to solve the puzzle(crossword) { for(i <- solutions.get){ //print all solutions to the crossword i.solvedCrossword foreach println } } } } -Thanks!! ^^ Why does getAllSolutions return an option? Is there a semantic di...