Gtjkyu

Spark's Column.isin function does not take List I am trying to filter out rows from my Spark Dataframe. val sequence = Seq(1,2,3,4,5) df.filter(df("column").isin(sequence)) Unfortunately, I get an unsupported literal type error java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(1,2,3,4,5) according to the documentation it takes a scala.collection.Seq list I guess I don't want a literal? Then what can I take in, some sort of wrapper class? 2 Answers 2 @JustinPihony's answer is correct but it's incomplete. The isin function takes a repeated parameter for argument, so you'll need to pass it as so : isin scala> val df = sc.parallelize(Seq(1,2,3,4,5,6,7,8,9)).toDF("column") // df: org.apache.spark.sql.DataFrame = [column: int] scala> val sequence = Seq(1,2,3,4,5) // sequence: Seq[Int] = List(1, 2, 3, 4, 5...

How to add the mongoDB driver and connector to the jupyter notebook? I would like to add the MongoDB Spark Connector v2.2 to the Jupyter notebook to pull data as dataframes from a mongodb database. However I keep on getting that error, I am not sure how to proceed from here. MongoDB Spark Connector v2.2 An error occurred while calling o40.load. : java.lang.NoClassDefFoundError: com/mongodb/ConnectionString The Spark version I am using is 2.3.1 2.3.1 The Python version I am using is 3.6.5 |Anaconda, Inc.| 3.6.5 |Anaconda, Inc.| I am running the Jupyter notebook on Windows 10 Windows 10 I downloaded the .jar files from the following link, specifically Version: 2.2.3-s_2.11 2.2.3-s_2.11 How to add the mongoDB driver and connector to the jupyter notebook ? I am not quite sure how to point the jupyter notebook to the correct location in windows 10 The code is as follows import findspark findspark.init() import os %env os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.mongod...

Start kubernetes pod memory depending on size of data job is there a way to scale dynamically the memory size of Pod based on size of data job (my use case)? Currently we have Job and Pods that are defined with memory amounts, but we wouldn't know how big the data will be for a given time-slice (sometimes 1000 rows, sometimes 100,000 rows). So it will break if the data is bigger than the memory we have allocated beforehand. I have thought of using slices by data volume, i.e. cut by every 10,000 rows, we will know memory requirement of processing a fixed amount of rows. But we are trying to aggregate by time hence the need for time-slice. Or any other solutions, like Spark on kubernetes? Another way of looking at it: How can we do an implementation of Cloud Dataflow in Kubernetes on AWS 2 Answers 2 It's a best practice always define resources in your container definition, in particular: r...

Under the hood SPARK Dataframe optimization Leaving aside the database connection aspects that get discussed with mapPartitions for RDDs, and noting that for me the Dataframe under the hood is harder to follow than the RDD abstraction: I didn't understand the first part. The answer that you are looking for 2nd question is partially available in this link stackoverflow.com/a/29012187/3213772 – puru Jul 2 at 8:59 You mean bullet 1: if so: mapPartitions is seen as a performance booster for RDDs. If DF's are so good, then how does the underhood work to equate to better than performance of an RDD using mapPartitions? – thebluephantom Jul 2 at 9:02 @puru But that li...