await a promise set in a pre navigation hook

Hi All,

I have a pre navigation hook that listens for requests and if they return images saves them to the cloud

  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud

  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud

This works in 95% of cases however there are some where the main request hander completes before each res.response()res.response() can be collected. This causes an error since the browser context closes with the main request handler. My question is how can i best get around this? One idea I had was that I could put a promise in the page userdata that resolves when the page has no outstanding images. After reading the docs however im not sure if this is possible since userdata needs to be serialisable? Has anyone else encountered this type of issue and how have they got around it?

Apify & Crawlee•2y ago•

3 replies

dual-salmon

await a promise set in a pre navigation hook

Hi All,

I have a pre navigation hook that listens for requests and if they return images saves them to the cloud

  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud

  return async (context) => {
    if (context.request.label == requestLabels.article) {
      context.page.on('request', async (req) => {
        if (req.resourceType() == 'image') {
          const response = await req.response();
          // extra processing and save to cloud

Continue the conversation

Join the Discord to ask follow-up questions and connect with the community

A&C

Apify & Crawlee

This is the official developer community of Apify and Crawlee.

21,256 Members

Join

await a promise set in a pre navigation hook

await a promise set in a pre navigation hook

Continue the conversation

Apify & Crawlee

Continue the conversation

Apify & Crawlee

Similar Threads