join streaming and static dataframes does nothing
Clash Royale CLAN TAG #URR8PPP join streaming and static dataframes does nothing So I'm trying to setup a stream from Kafka, and then add columns from a table to that dataframe based on a matching id in the kafka message and table. I've got two separate files, the first one calls a method in streamKafka_ASH to setup the stream and then it prints out the contents. val query = streamKafka_ASH.setupStream(spark, bootstrapservers,zkPath, topics, zkHosts, consumerGroup, batchIntervalSeconds.toInt, autoOffset, checkpointDir) .writeStream .outputMode("append") .format("console") .trigger(ProcessingTime(intervalMs = batchIntervalSeconds.toInt * 1000)) .start() query.awaitTermination() And here is the code in streamKafka_ASH.setupStream(...) val marketingTrackingSchema = new StructType() .add("id", StringType) .add("device", StringType) .add("type", StringType) .add(...