Multi tool use

Clash Royale CLAN TAG

Extracting data from godaddy using jsoup

I'm using Jsoup to extract html from Godaddy's website. I want to extract this specific segment below. I have both the final webpage's specific segment, where it states "Sorry, google.com is taken" and the HTML code itself.

However in my program I have the following:

import java.io.IOException;


import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class test {
public static void main(String args) throws IOException {
String url = "https://www.godaddy.com/dpp/find?checkAvail=1&tmskey=&domainToCheck=google";
Document document = Jsoup.connect(url).get();
Element div = document.getElementById("searchResults");
Elements spans = div.select("span");
for (Element e: spans)
System.out.println(e.text());
}
}


However, this code prints a NullPointerException. I know that JSoup cannot work with JS, but this is HTML and for some reason it is not being extracted. I also tried extracting all the HTML from the page and it does not contain these words.

Can anyone point me in the right direction or give me another way to extract this piece of information from godaddy?

1 Answer
1

First of all the url you have provided redirects to some other location so you will need to follow redirects:

Document document = Jsoup.connect(url).followRedirects(true).get()


But even that will not fix your problem. To show if a domain is available or not the website uses javascript to fetch data from the server. Now this request will fail since this is coming from a unknown origin.

In short Pedro is right you have to use the API.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.