Utilizing rvest to extract user videos from Twitter

When using rvest to extract data from websites, I found that I am unable to scrape dynamic content. For instance, how can I extract the audience count (44K) from this video post?

https://i.sstatic.net/CRXZ9.png

Here's what I attempted:

library(rvest)
video_tweet = html("https://twitter.com/estrellagalicia/status/993432910584659968")
video_tweet %>% 
  html_nodes("#permalink-overlay #permalink-overlay-dialog div #permalink-overlay-body div div div div div div div div div div div div div span div div div div span span") %>% as.character()
cssrtwitterweb-scrapingrvest

Answer №1

To successfully scrape data from Twitter, you will need to utilize Rselenium and select the appropriate CSS. Here is a snippet of code that should help you achieve this:

library(RSelenium)
library(rvest)
rmDr <- rsDriver(browser = "chrome")
myclient <- rmDr$client
video_tweet = "https://twitter.com/estrellagalicia/status/993432910584659968"

myclient$navigate(video_tweet)

mypagesource <- myclient$getPageSource()
read_html(mypagesource[[1]]) %>% 
    html_nodes("#permalink-overlay-dialog > div.PermalinkOverlay-content > div > div > div.permalink.light-inline-actions.stream-uncapped.has-replies.original-permalink-page > div.permalink-inner.permalink-tweet-container > div > div.js-tweet-details-fixer.tweet-details-fixer > div.card2.js-media-container.has-autoplayable-media > div.PlayableMedia.LiveBroadcastCard-playerContainer.LiveBroadcastCard--supportsLandscapePresentation.watched.playable-media-loaded > div > div > div > div:nth-child(2) > div.rn-1oszu61.rn-1efd50x.rn-14skgim.rn-rull8r.rn-mm0ijv.rn-13yce4e.rn-fnigne.rn-ndvcnb.rn-gxnn5r.rn-1nlw0im.rn-deolkf.rn-6koalj.rn-1pxmb3b.rn-7vfszb.rn-eqz5dr.rn-1r74h94.rn-1mnahxq.rn-61z16t.rn-p1pxzi.rn-11wrixw.rn-ifefl9.rn-bcqeeo.rn-wk8lta.rn-9aemit.rn-1mdbw0j.rn-gy4na3.rn-u8s1d.rn-1lgpqti > span > div > div.rn-1oszu61.rn-1efd50x.rn-14skgim.rn-rull8r.rn-mm0ijv.rn-13yce4e.rn-fnigne.rn-ndvcnb.rn-gxnn5r.rn-deolkf.rn-6koalj.rn-1pxmb3b.rn-7vfszb.rn-eqz5dr.rn-1mnahxq.rn-61z16t.rn-p1pxzi.rn-11wrixw.rn-ifefl9.rn-bcqeeo.rn-wk8lta.rn-9aemit.rn-1mdbw0j.rn-gy4na3.rn-bnwqim.rn-1lgpqti > div > div > span > span") %>% as.character()

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Unable to adjust price slider on mobile device, bars are not sliding

I have been utilizing the jQuery slider from lugolabs.com for my website, and while it works flawlessly on desktop, it seems to have issues when viewed on mobile devices. You can check out the slider at this link: The library used for this slider is jQue ...

javascriptjqueryhtmlcssangularjs

Issue with reordering columns in Bootstrap 5

Currently, I am transitioning to Bootstrap 5 and facing challenges with the column re-ordering feature. Within my layout, I have three columns: a title, an image, and a paragraph of text. My goal is to display the image on the left, the title on the right, ...

cssbootstrap-5

Creating a Cleveland dot plot using ggplot2

I recently stumbled upon a report from TIMSS which featured a plot (displayed below) that I found to be highly informative. This particular type of plot is known as a Cleveland dot plot, but it also includes confidence intervals. I am curious to know if ...

pythonrmatplotlibplotggplot2

Prevent scrollbar from appearing while splash page loads

Looking for help with a script to create a splash/intro page loader. $(function(){ setTimeout(function() { $('#splash').fadeOut(500); }, 6000); }); The current script hides the intro page after 6 seconds, ...

javascriptjqueryhtmlcss

Jquery Toggle not applying class change on mobile devices

Check out my website: . There's a menu option called Services with a dropdown feature. Here is the code for it: <li class="dropdown menu-services"><a class="dropdown-toggle" href="#" data-toggle="dropdown" data-target="#">Services <b c ...

jquerycss

Integrate a resizable sidebar on the webpage's right side that dynamically adjusts the layout

As I develop a Chrome Extension, my goal is to incorporate a sidebar into all webpages without overlapping the existing content. The sidebar should be placed beside the content, effectively reducing the width of the body of the webpage to the initial width ...

javascripthtmlcssgoogle-chrome-extensionsidebar

Fresh items for the table in the regression loop

Hello, I am starting out with R and I am attempting to run regressions by group. My dataset consists of 13 groups, and I aim to generate 13 separate regression results—one for each group. My goal is to compile all these results into a table. Here is the ...

rloops

Modify CSS using Javascript by targeting the class of ".modalDialog > div {}"

I am currently working on a project that involves creating a Modal Layer using CSS. I have successfully implemented the modal layer, but now I need to figure out how to change the width from the default 400px to something else when I open it with Javascrip ...

javascriptcss

`How can you indicate a clicked linkbutton from a group of linkbuttons in asp.net?`

There is a single aspx page with a set of link buttons on it. Link Button 1 Link Button 2 Link Button 3 Link Button 4 Link Button 5 When any of the link buttons are clicked, they should be highlighted. All these link buttons are contained within a table ...

.netasp.netcss

Move the button text back to its original position with animation

I have a button with text that shifts to the right by 15px when hovered over. However, when the cursor is removed, the text immediately snaps back instead of smoothly transitioning back to its original position. I tried setting the ease-in-out property but ...

htmlcss

Image link in HTML / CSS not functional on one half of the image

My webpage has an image thumbnail with accompanying text positioned to the right. I have a hyperlink on the image that should open the full-size version when clicked. However, there seems to be an issue where the hyper link only activates on the lower hal ...

htmlcss

Is it possible to create varied background shades in CSS3 without relying on images?

Check out this fiddle: http://jsfiddle.net/dzAJp All the divs are styled with a light blue background color using the background-color CSS property instead of background-image. I want to darken the second div slightly by combining its current background ...

css

Json Swagger Editor: Error - Undefined

There is an undefined error in rendering when using the Swagger Editor. Here is the code snippet: swagger: '2.0' info: version: 1.0.0 title: AccountBalance host: localhost:2625 basePath: /accountbalance schemes: - http - https consumes: ...

cssfacebookspring

Adjust a Javascript script to choose the best font color for use in R Shiny applications

I am currently seeking to determine the font color of hover messages based on the background color. This means white if the background is dark, and black if it is light. However, I stumbled upon a Stack Overflow question with a Javascript solution that see ...

javascriptrshiny

Positioning an element in the center of another using JQuery

Greetings! I am currently working with some elements that look like this: <div id="one">content</div> <div id="two">content</div> These elements are followed by another set of elements (without any parent, placed directly after th ...

javascriptjquerycsspositionpositioning

The Impreza theme is experiencing difficulty displaying Font Awesome icons

Recently, I had a client who needed a WordPress website. After working on the project on my own server, I finally completed it and transferred the entire site to the client's main domain at gulugalmarketing.com. Everything was functioning perfectly o ...

htmlcsswordpressfont-awesome

Adjusting the width of row items in Angular by modifying the CSS styles

I am envisioning a horizontal bar with items that are all the same width and evenly spaced apart. They can expand vertically as needed. Check out the updated version here on StackBlitz https://i.sstatic.net/MFfXd.png Issue: I am struggling to automatica ...

cssangulartypescript

Header with absolute positioning doesn't play nice when nudged in Chrome

Take a look at the HTML page on this JSFiddle link: http://jsfiddle.net/rb8rs70w/2/ The page features a "sticky" header created using CSS with the property position: absolute. ... <body class="body-menu-push"> <header role="banner"> ...

htmlcssgoogle-chrome

Node.js request package automatically rounds large integers before returning them

I am currently utilizing the npm package request for making calls to the Twitter API. Here is an example of how I am doing so: request({ url: `https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=${screenName}&${countQuery}&tweet_mo ...

node.jstwitterhttprequestnpmjs

Challenges arise with the safari navigation bar within a PWA platform

Struggling with the formatting and spacing issues of a PWA application on IOS devices due to notches and navbar heights. One specific problem is that a chat input with absolute positioning and bottom: 0 does not display properly in Safari because of the n ...

htmlcssprogressive-web-apps