Ciugk

Clash Royale CLAN TAG #URR8PPP What is the semantic relationship expected between word vectors which are scalar multiples of each other in word2vec? Let's say you have a word vector for the word queen . Some of its scalar multiples would be x = queen + queen , y = queen + queen + queen and n * queen for any real value of n ( so we're also considering non-integer values of n such as in 0.83 * queen ). queen x = queen + queen y = queen + queen + queen n * queen 0.83 * queen Therefore I have two questions: How are these points expected to appear on the biplot generated after PCA? Will they lie around a straight line? What is the semantic relationship expected between the words x , y and queen ? I know these vectors will all have the same ratio between the dimensional values within the vector, but I'm having a hard time figuring out how to read this in terms of word meaning. x y queen By clicking "Pos...

Clash Royale CLAN TAG #URR8PPP text similarity distinct method using spark？ I want to get a text similarity distinct method on 200 million differnent sentences using spark .Suppose I have 4 sentence that is ["Hi I heard about Spark","Hi I heard about Spark World", "Logistic regression models ","Logistic regression goodmodels "] I hope get the reuslt is ["Hi I heard about Spark", "Logistic regression models] Since the first sentence is similar to second sentence and the third sentence is similar to the 4th sentence arrorcding to Levenshtein distance:https://rosettacode.org/wiki/Levenshtein_distance How to achieve it efficiently using spark? Because the data is 200 million, I am hesitate to do cartesian By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your con...

Clash Royale CLAN TAG #URR8PPP RegularExpression: match and extract long domain I want to match and extract domain form strings,and I got an equation: result = re.findall(r"(^((?!-))(xn--)?[a-z0-9][a-z0-9-_]{0,61}[a-z0-9]{0,1}.(xn--)?([a-z0-9-]{1,61}|[a-z0-9-]{1,30}.[a-z]{2,})$)", text) It does well for domain like : example.org example.org.eu but it cannot work for domain like : sub_example.example.org.eu so,I am asking for helping improving the equation. By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Clash Royale CLAN TAG #URR8PPP NLTK Named Entity Recognition with Custom Data I'm trying to extract named entities from my text using NLTK. I find that NLTK NER is not very accurate for my purpose and I want to add some more tags of my own as well. I've been trying to find a way to train my own NER, but I don't seem to be able to find the right resources. I have a couple of questions regarding NLTK- I would really appreciate help in this regard 4 Answers 4 Are you committed to using NLTK/Python? I ran into the same problems as you, and had much better results using Stanford's named-entity recognizer: http://nlp.stanford.edu/software/CRF-NER.shtml. The process for training the classifier using your own data is very well-documented in the FAQ. If you really need to use NLTK, I'd hit up the mailing list for some advice from other users: http://groups.google.com/group/nltk-users. ...

Clash Royale CLAN TAG #URR8PPP spacy: adding a comma changes the dependency tree I have noticed that this particular comma changes the dependency tree in a big way: London broke the world records, and NY will be huge ________________ London broke the world records and NY will be huge ___________ With the comma, the right subtree is "NY will be huge" and without it, it is just "will be huge". It seems to me that "NY will be huge" should be the correct subtree. I am new with NLP and spacy, so not really sure: Should I report this as a bug or are there other tools to help in these kind of situations? spacy version '2.0.11' By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subje...

Clash Royale CLAN TAG #URR8PPP Picking groups of sentences by topics and features My plan is to vectorise all these sentences (with glove for example). Start at sent #1; add sentences until none of the distances improve. Repeat with sent #2. At the end, I could have top 5/10 groups of sentences for each topic. Unfortunately, I'm not taking into consideration my binary features. Do I concatenate them to my word vectors and use them for computing distance? Do I train a multi-class supervised model (three topics and other?) with a small imbalanced dataset where I classify each sentence separately. Use this model to make predictions on each sentence and build groups based on average predictions? I don't like this way. Or do I stage this as two problems? Stage 1, get all the distances for each potential group of sentences with a certain minimum threshold. Stage 2 classify each group on these binary features with the topic similarities? Can somebody point me in the right direction? ...