Currently utilizing the Ruby Selenium web driver, my goal is to extract content from the Google Knowledge Graph located on the top right-hand side of the search results page. This element is within the <div class="xpdopen"> section.
@driver = Selenium::WebDriver.for :phantomjs
@driver.manage.timeouts.implicit_wait = 10
@driver.get "http://google.com"
element = @driver.find_element :name => "q"
element.send_keys "BMW"
element.submit
content = @driver.find_element(:class, 'xpdopen')
Despite my efforts, Selenium fails to locate this specific element and throws an error.
#<Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with class name 'xpdopen'"
Interestingly, when testing in a Chrome JavaScript console using $('.xpdopen'), the element is found immediately.
I have also attempted:
@driver.execute_script("return document.getElementsByClassName('xpdopen');")However, this method also does not locate the desired element.
Even after trying @driver.page_source, it appears that <div class="xpdopen"> is not present in the page source, despite being visible in the Chrome console. Why is this happening?
How can I successfully retrieve this element using Selenium?
Below are the results obtained from my testing in pry:
[21] pry(main)> @driver = Selenium::WebDriver.for :phantomjs
=> #<Selenium::WebDriver::Driver:0x..f822d288ec7f0a708 browser=:phantomjs>
[22] pry(main)> @driver.manage.timeouts.implicit_wait = 10
=> 10
[23] pry(main)> @driver.get "http://google.com"
=> {}
[24] pry(main)> element = @driver.find_element :name => "q"
=> #<Selenium::WebDriver::Element:0x..f389f4a8876f601e id=":wdc:1434526425103">
[25] pry(main)> element.send_keys "BMW"
=> nil
[26] pry(main)> element.submit
=> {}
[27] pry(main)> sleep 10
=> 10
[28] pry(main)> content = @driver.find_element(:xpath, '//*[@id="rhs_block"]/ol/li/div[1]/div')
Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with xpath '//*[@id=\"rhs_block\"]/ol/li/div[1]/div'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"gzip;q=1.0,deflate;q=0.6,...
Additionally,
[29] pry(main)> content = @driver.find_element(:css, "#rhs_block > ol > li > div.kp-blk._Jw._Rqb._RJe > .xpdopen")
Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with css selector '#rhs_block > ol > li > div.kp-blk._Jw._Rqb._RJe > .xpdopen'","request":{"headers":{"Accept":"application/json","Accept-...
Lastly, to showcase that other elements on the same page are being identified without issues:
[30] pry(main)> results = @driver.find_elements(:xpath, "//p/a")
=> [#<Selenium::WebDriver::Element:0x6f6a74631e2b7010 id=":wdc:1434527087873">, ...]
Following a screenshot comparison, it was discovered that PhantomJS does not render the content of the Knowledge Graph.
Screenshot from PhantomJS
Screenshot from Firefox
Why is there no content from the Knowledge Graph displayed in PhantomJS?