I have developed a website that utilizes a user's webcam frames through an HTML <video> element. These frames are then copied to a <canvas> element, which is displayed using opencv.js' cv.imshow(). Additionally, the frame is duplicated onto an off-screen canvas for server-side processing, producing the intended outcome.
The website performs as expected on Chrome (including browsers like Brave) on desktop (Linux) and Android Chrome. However, there seems to be an issue with iOS Chrome and Safari where the canvas appears as plain white rather than displaying the video frames. This discrepancy verifies that the camera frames from the video element are being captured accurately.
In order to troubleshoot this problem, I implemented the following functions:
To check if the canvas is blank:
// Returns true if all color channels in each pixel are 0 (or "blank")
function isCanvasBlank(canvas) {
return !canvas.getContext('2d')
.getImageData(0, 0, canvas.width, canvas.height).data
.some(channel => channel !== 0);
}
To verify the visibility of the canvas element:
const observer = new IntersectionObserver((entries) => {
if(entries[0].isIntersecting){
// The element is visible
console.log('canvas_visible', true);
} else {
// The element is not visible
console.log('canvas_visible', false);
}
});
observer.observe(canvas);
Upon inspecting chrome://inspector on iOS, the output displays is_canvas_blank false and canvas_visible true after each frame.
An interesting observation is that running the opencv.js tutorial on iOS Chrome successfully displays the frames on the canvas (refer to the opencv.js videocapture tutorial). My own implementation closely follows the steps outlined in this tutorial (see minimal implementation below).
HTML
<div id="localVideo">
<video id="video"></video>
<canvas id="canvas" height="480px" width="640px"></canvas>
<canvas id="hiddenCanvas" height="480px" width="640px"></canvas>
</div>
JS
let video = document.getElementById('video');
let cap = new cv.VideoCapture(video);
let src = new cv.Mat(video.videoHeight, video.videoWidth, cv.CV_8UC4);
let dst = new cv.Mat(video.videoHeight, video.videoWidth, cv.CV_8UC1);
const FPS = 15;
function processVideo() {
try {
if (!streaming) {
// Cleanup and stop.
delete cap;
src.delete();
dst.delete();
}
cap.read(src);
cv.cvtColor(src, dst, cv.COLOR_RGBA2RGB);
cv.imshow('canvas', dst);
cv.imshow('hiddenCanvas', src);
let delay = 1000 / FPS;
setTimeout(processVideo, delay);
} catch (err) {
console.log(err);
}
}
// Begin processing the video.
setTimeout(processVideo(), 0);
The HTML <video> element also correctly displays the video on iOS Chrome.
If any frontend developer could shed light on why this issue pertains only to iOS Chrome/Safari, why the tutorial functions whereas my code does not, or if I am overlooking anything, it would be greatly appreciated. Thank you.
Screenshots from iOS Chrome are attached for reference:
Initial page load - Canvas with the image source set to PNG displaying correctly:
https://i.sstatic.net/snJNe.jpg
Displaying the video element showing the user's cam - operating as expected:
https://i.sstatic.net/6nwcv.jpg
Video element hidden and canvas element unhidden - no rendering on the canvas
https://i.sstatic.net/ot3I3.jpg
Expected result (as observed on Android Chrome):
https://i.sstatic.net/iJ0Hc.jpg