Nnursing-lime
Best Crawler for Youtube JS?
I'm looking to scrape youtube captions. There is a timedtext url in the page js... but it doesn't seem to show through standard http request through curl etc.
Can you please suggest a method?
Here is my routes.ts file:
import { Dataset, createPuppeteerRouter } from 'crawlee';
export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ request, page, log }) => {
log.info(
// Wait for the page to fully load
await page.waitForSelector('ytd-watch-flexy', { timeout: 10000 }); // Adjust the selector and timeout as necessary
log.info(
// Execute JS in the page context to find the caption URL
const captionUrl = await page.evaluate(() => {
const scripts = Array.from(document.querySelectorAll('script'));
for (const script of scripts) {
const match = script.textContent?.match(/https://www.youtube.com/api/timedtext?[^"]+/);
if (match) {
return match[0];
}
}
return null;
});
if (captionUrl) {
log.info(
await Dataset.pushData({
url: request.loadedUrl,
title: await page.title(),
captionUrl,
});
} else {
log.warning(
}
});
Can you please suggest a method?
Here is my routes.ts file:
import { Dataset, createPuppeteerRouter } from 'crawlee';
export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ request, page, log }) => {
log.info(
Default handler triggered for: ${request.url}// Wait for the page to fully load
await page.waitForSelector('ytd-watch-flexy', { timeout: 10000 }); // Adjust the selector and timeout as necessary
log.info(
Page is ready for processing: ${request.url}// Execute JS in the page context to find the caption URL
const captionUrl = await page.evaluate(() => {
const scripts = Array.from(document.querySelectorAll('script'));
for (const script of scripts) {
const match = script.textContent?.match(/https://www.youtube.com/api/timedtext?[^"]+/);
if (match) {
return match[0];
}
}
return null;
});
if (captionUrl) {
log.info(
Found caption URL: ${captionUrl}await Dataset.pushData({
url: request.loadedUrl,
title: await page.title(),
captionUrl,
});
} else {
log.warning(
No caption URL found on ${request.loadedUrl}}
});
This is the official developer community of Apify and Crawlee.
14,091Members
Resources
Recent Announcements
Similar Threads
Was this page helpful?
