Ciugk

Clash Royale CLAN TAG #URR8PPP text similarity distinct method using spark？ I want to get a text similarity distinct method on 200 million differnent sentences using spark .Suppose I have 4 sentence that is ["Hi I heard about Spark","Hi I heard about Spark World", "Logistic regression models ","Logistic regression goodmodels "] I hope get the reuslt is ["Hi I heard about Spark", "Logistic regression models] Since the first sentence is similar to second sentence and the third sentence is similar to the 4th sentence arrorcding to Levenshtein distance:https://rosettacode.org/wiki/Levenshtein_distance How to achieve it efficiently using spark? Because the data is 200 million, I am hesitate to do cartesian By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your con...

Clash Royale CLAN TAG #URR8PPP Spark - Why is it necessary to collect() to the driver node before printing an RDD? Can it not be done in parallel? I was reading about how to print RDDs in Spark (I'm using Java), and it seems like most people just collect() (if the RDD is small enough) and use forall(println), or something like that. Is it not possible to print in parallel? Why do we have to collect the data onto the driver node in order to print? collect() I was thinking maybe it's because we can't use System.out in parallel, but I feel like that's not it. And furthermore, I'm not quite sure how one would even distribute the data and print parallelly, in terms of code. One approach I was thinking of was to do a mappartitions that doesn't do anything useful in terms of mapping, but it iterates through the partition and prints its contents. By clicking "Post Your Answer", you acknowledg...