Can Rvest be used to extract data from an HTML node while excluding child classes?

I need assistance with extracting data from a specific URL 

(https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine)
. The posts on this page contain replies that start with "Originally Posted by ...". I want to scrape all the content within the posts while excluding the initial "Originally posted by" text. Here is an example:

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Originally posted by C Heuwi 
      Hellou
 E    Hello guys
 F    Originally posted by A Hi, how are you ?
      I am doing good
 G    Whats going on ?

The required information for post by user D is "Hellou", which is found under div.quote_container class (child class), and "I am doing good" is under blockquote.postcontent.restore, its parent class.

Expected output:

User  df_text
 A    Hi, how are you ?
 B    This is beautiful!
 C    Heuwi
 D    Hellou
 E    Hello guys
 F    I am doing good
 G    Whats going on ?

I have tried several methods as shown below:

url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)
threads<- cbind(review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)") %>% html_text())

Additionally, I experimented with these alternatives:

threads <- cbind(review %>% html_nodes(xpath = '//div[@class="blockquote.postcontent.restore"]/node()[not(self::div)]') %>% html_text())

or

threads <- review %>% html_nodes(".content")
close_nodes <- threads %>% html_nodes(".quote_container")
chk <- xml_remove(close_nodes)

Unfortunately, none of these approaches yielded the desired outcome. Any suggestions on how to successfully extract the post data while excluding the child class would be greatly appreciated. Thank you in advance!

htmlcssrweb-scrapingrvest

Answer №1

One way to tackle this issue is by leveraging the xml_remove function provided in the xml2 library, which gets automatically loaded with rvest.

library(rvest)
#fetch page data
url<-"https://forums.vwvortex.com/showthread.php?8829402-Atlas-V6-Oil-Change-Routine"
review <- read_html(url)

#locate parent nodes
threads<- review %>% html_nodes("blockquote.postcontent.restore:not(.quote_container)")
#identify child nodes for exclusion
toremove<-threads %>% html_node("div.bbcode_container")
#eliminate specified nodes
xml_remove(toremove)

#convert parent nodes to text format
threads %>% html_text(trim=TRUE)

The documentation for xml_remove warns "Care needs to be taken when using xml_remove()". Always proceed with caution and remember to save your work regularly.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The challenge of using CSS Flexbox to position the logo with a margin-top

While learning flexbox, I encountered a challenge. I aim to center align h1, p, and button elements, with the logo positioned at the top margin-top of 1em. Essentially, I want the h1, p, and button to be centered, while the logo is positioned at the top wi ...

htmlcssflexbox

What causes a custom element to suddenly crumble?

I am attempting to create a custom field that can display either an SVG or a Canvas, but the rendering is not as expected. I anticipate two boxes that are 400 pixels wide and 300 pixels high, however, they appear to collapse in an unusual manner. How can I ...

htmlsvgecmascript-6html5-canvas

Unable to establish a connection with localhost on port 3918 due to a connection refusal error in R Rselenium

I am in need of assistance in resolving an error related to the subject line. Despite reviewing and attempting the solutions offered in previous posts, the issue remains unresolved. Historically, updating R and RStudio has resolved this same error. The ...

rrseleniumnetstatrsdriver

What is the method for calculating the effect size [90%CI] and including it in the summary table using the R package "gtsummary

I've been experimenting with creating a summary table using the R package "gtsummary", and I must say, it offers a lot of flexibility. The add_stat function provides the option to include add-ons, such as informing the effect size with confidence inte ...

rsizeeffectgtsummary

How can elements be styled depending on the flex-wrap state?

Is it possible to determine if an element has been wrapped due to the use of flex-wrap, and style it accordingly? This is a fairly simple concept that can have a big impact on the presentation of your content. ...

csscss-selectorsflexbox

What is the best way to sort rows based on the larger value in a separate column?

I currently have a data frame structured as shown below: d1<-c('a','b','c','d','e','f','g','h','i','j','k','l') d2<-c(1,5,1,2,13, ...

rdataframedplyrdata-wrangling

Position three divs in the center and at the top of my master page

My goal is to position 3 divs on top of each other efficiently without wasting space. The first div contains a logo image, the second is a search box, and the third, located at the right side, is a login/logout control. I have attempted to use CSS to floa ...

javascriptcsshtml

Executing a webservice method in an html page using javascript without the need to refresh the page

Is it possible to call a webservice from an index.html page using JavaScript? My webservice is located at "localhost/ws/service.asmx" and the specific web method I want to call is called HelloWorld. The index.html page contains an HTML submit button whic ...

javascriptjqueryhtmlweb-services

What are some ways to create a table that can be easily filled in?

I'm striving to enhance the user experience by allowing a table cell to be easily editable with just a double click, converting it into an input field with the existing cell value pre-populated. Currently, I have successfully achieved this functional ...

htmlangularjsuser-interfacehtml-table

The website functions perfectly on a local server, but once it's uploaded, the images fail

My website is functioning properly on my computer, however, after uploading it to the server, I noticed that some of the links are not appearing. Upon inspecting these links through developer tools, it shows an image size of 'Natural 1 X 1' despi ...

htmlcss

Default Documents on Login.aspx are being displayed even though it hasn't been specified in the web.config file

When I decided to move my website from the built-in web server (Cassini) to IIS, I encountered an issue with my Login.aspx page. The web.config file was showing an error "The requested page cannot be accessed because the related configuration data for the ...

asp.netdefaulthtml

Navigating the FormSpree redirect: Tips and tricks

I recently set up my website on Github Pages and wanted to integrate a free contact form from FormSpree. However, I encountered an issue where after submitting the form, it redirected to a different website, which was not ideal. After researching online, I ...

javascripthtmlforms

What is the best way to address the problem of quotes while utilizing Freemarker in conjunction with ckEditor (or any other html editor

Consider this scenario within my template: ${(object.attribute)!"default text"} At times, quotation marks are necessary, such as when FreeMarker is handling null dates and similar instances. The issue arises when my HTML editor automatically transforms ...

javahtmltemplatesckeditorfreemarker

function for clicking on mobile navigation bars

I could use a bit of assistance with understanding this JavaScript function. I am trying to create an onclick function for the mobile navigation "li" that will cause it to slide up. Can someone help me figure this out? Thank you. <!-- /#js --> < ...

javascriptjqueryhtml

Why does the selected attribute not function properly in <select><option> when used with ngModel?

I encountered an issue with Angular 6: when using a component based on the <select> element (combo-box), everything works fine if I use it in the traditional way, where I specify the selected attribute. In this case, the default option appears select ...

htmlangular6html-selectangular-ngmodel

Automated Web Scraping with Selenium: Utilizing the Next Button

I am currently attempting to extract press releases from the Federal Reserve's official website at https://www.federalreserve.gov/newsevents/pressreleases.htm. In order to access documents from past years, I need to navigate to the next page by clicki ...

pythonseleniumselenium-webdriverweb-scrapingscrape

Displaying an array value instead of a new value list in a React component

Situation - Initial number input in text field - 1 List of Items - 1 6 11 Upon removing 1 from the text field, the list becomes - New List Items - NaN NaN NaN Now, if you input 4 in the field. The updated List Items are - NaN NaN 4 9 14 Expected ...

javascripthtmlcssreactjsreact-usecallback

Substitute common words in a list-based column

Here is a scenario with a data frame: test = data.frame(language=c("german", "english"), text=I(list(c("und das Beil", "wichtige Thematik der"), c("some useful information", "the most unuseful product")))) The task at hand involves removing stopwords fro ...

rapply

JavaScript tabs function malfunctioning upon page load

I attempted to implement tabs on my website using this example: http://www.w3schools.com/howto/howto_js_tabs.asp Although everything is functioning correctly, the default tab isn't opening as expected. The tab opens, but the header color does not get ...

javascriptjqueryhtml

Finding the R Data Frame Value using Search

I need assistance with performing a lookup in R to return values for a specific variable from a data frame. For instance: I have a variable called TaskName and I am seeking guidance on how to retrieve dates and calculate the number of days between those ...

rdataframe