What is the most effective way to extract data from a .dpbox table using selectorgadget in R (rvest)?

Recently, I've been experimenting with web scraping data from different websites using selectorgadget in R. One successful example was when I extracted information from . My usual approach involves utilizing the selectorgadget Chrome extension to choose the tables I need and then inserting the CSS Selection outcome into my code like this:

urlx <- "http://www.dotabuff.com/heroes/abaddon/matchups"
rawData <- html_text(html_nodes(read_html(urlx),"td:nth-child(4) , td:nth-child(3), .cell-xlarge"))

When trying to extract data from , my selectorgadget query looked like this:

urlx <- "http://www.dotapicker.com/heroes/abaddon"
rawData <- html_text(html_nodes(read_html(urlx),".ng-scope:nth-child(1) .ng-scope .ng-binding"))

However, this time, no nodes were returned after calling the html_nodes function, resulting in:

{xml_nodeset (0)}

I suspect that the issue might be related to the structure of the table being nested within a drop-down box, unlike the previous scenario where the table was directly on the webpage. I'm currently exploring solutions to overcome this challenge.

Your assistance is greatly appreciated!

Answer №1

Upon examining this website, it appears that data is loaded dynamically through XHR requests. To confirm this in Chrome, you can navigate to the inspect tool and access the network tab. Once there, you will observe various json files being fetched. By directly scraping these json files, you can then extract the information you require. Below is a brief illustration:

library(httr)
library(jsonlite)

heroinfo_json <- GET("http://www.dotapicker.com/assets/json/data/heroinfo.json")
heroinfo_flat <- fromJSON(content(heroinfo_json, type = "text"))
#> No encoding supplied: defaulting to UTF-8.

winrates_json <- GET("http://www.dotapicker.com/assets/dynamic/winrates10d.json")
winrates_flat <- fromJSON(content(winrates_json, type = "text"))
#> No encoding supplied: defaulting to UTF-8.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Angular showcases the presence of a signed-in user at page load, disregarding the absence of

Currently, I am following a course where I have implemented a simple login form inside the navigation bar: <nav class="navbar navbar-expand-md navbar-dark fixed-top bg-primary"> <div class="container"> ...

What is the significance of using "your-shop" as the action form without a file extension?

I've come across filenames (with file extensions) or URLs inside the action attribute of a form, but I have never seen code like this before: <form action="your-shop" name="shop_name_form" id="shop_name_form" method="post" onsubmit="return check_s ...

"Encountering a glitch while attempting to upload documents into RavenDB

I am attempting to store attachments in RavenDB, but I keep encountering a "file not found" error. Here is the code snippet from my MVC View: <input type="file" name="file" id="Ids2" style="float:right"/> Through an AJAX call, I send the selected ...

Converting text/plain form data to JSON using Node.js - step by step guide

I am currently working on a Node.js application to execute a POST call to an API for placing an order. However, the data I receive in our app is in text/plain format and not JSON. This is the current format of the data: TypeOrder=buy Coin=BTC AmountCoin= ...

Updating an HTML Table with AJAX Technology

I'm struggling to figure out how to refresh an HTML table using AJAX. Since I'm not the website developer, I don't have access to the server-side information. I've been unable to find a way to reload the table on this specific page: I ...

Tips for programmatically choosing dropdown menus in Angular 5

Trying to set up a dropdown select option in Angular 5 that dynamically changes based on the backend's input. https://i.sstatic.net/8cgsh.png Keep in mind that both select boxes are identical, only the options inside them vary. selector.html <h ...

How to use AngularJS to collapse various panels with unique content

Hey everyone, I'm working on developing a collapsible panel using Angular. The panel should consist of a header and body to display the content. The desired behavior is that when a button is clicked, the content collapses down, and clicking the same b ...

Using HTML and CSS to implement a broadened perspective for a specific design

https://i.stack.imgur.com/ckQHa.png Hello, I am facing an issue with a UX design that is optimized for 1200px resolution width. However, when the HTML is loaded in a browser on a larger window resolution, there is a 200px gap on the right side. How can I ...

Maintain the HTML font color when printing - Issue with IE settings, not the printer

I've been struggling with this issue all day. Initially, I thought it was a problem with the print settings, but then I realized that it's actually the "Print background colors and images" option in IE causing the trouble. Here is the last test ...

Having trouble closing the phonegap application using the Back Button on an Android device

I've encountered an issue with my code for exiting the application. It works perfectly the first time, but if I navigate to other screens and then return to the screen where I want to close the app, it doesn't work. <script type="text/javascr ...

adding <script> elements directly before </body> tag produces unexpected results

While following a tutorial, the instructor recommended adding <script> tags right before the </body> to enhance user experience. This way, the script will run after the entire page content is loaded. After implementing the code block as sugges ...

Unable to load JQuery from a div element

My goal is to create a standard .html file containing the navigation, footer, and other elements that will be used across multiple pages for a small site I'm building. I want to keep it simple and avoid using php or other programming languages. I&apo ...

How can I transform an HTML div into a video file like MP4 using Python and Django?

I'm looking to take a HTML page and target a specific <div> within it in order to convert it into video format. Purpose: I understand that HTML is typically static, but I have a requirement to transform it into a video. I'm seeking method ...

Adjust the font color in code function for Woocommerce

I am looking to customize the appearance of a specific part of this code: '% discount' This section displays the percentage amount followed by the word "Discount," and I would like to apply a unique color to this "% discount" * Modify the disp ...

The side modal features a full-height design with headers, bodies, and footers that are

I have created a side modal that extends to full height. The content in the modal body is quite lengthy, but I am facing an issue where the header, body, and footer are not equal in size. I am considering using scrollspy on the HTML body tag to potentially ...

"Can anyone provide guidance on how to initiate a css 3d animation by clicking a button

Currently, I am developing a folding hide/show animation that can be triggered using Javascript. If you would like to take a look at the code and see a working example, please visit this link: You can also view just the gist here: https://gist.github.com ...

`The header navigation is not responding to window resizing functionality`

I am currently developing a header navigation that consists of a logo on the left side, a profile icon on the right side, and some navigation links in the middle. A left slide menu has been implemented to trigger when the window width is less than 700px. ...

Update dynamically generated CSS automatically

Is there a way to dynamically change the CSS? The problem I'm facing is that the CSS is generated by the framework itself, making it impossible for me to declare or modify it. Here's the scenario at runtime: https://i.sstatic.net/IovGr.png I a ...

Typescript includes empty spaces in its duplicate-checking process

I have been working on removing duplicate values from an array using the following code: for (var i = 0; i < a.length; i++) obj[a[i]] = a[i] a = new Array(); // Checking each object with keys to remove duplicates. for (var key ...

What is the best way to make an element "jump to the top" when the parent element has reached its maximum height?

I have a parent div containing 6 child divs. The height of the internal divs changes dynamically based on the data. The outer div has a fixed height set. I want the internal divs to move to a new column inside the parent div when they no longer fit in term ...