loud-coral•3w ago
@Kacka H. @Lukas Krivka @vladdy @
@Kacka H. @Lukas Krivka @vladdy @Ella Greetings dear collegium, I'm trying to use my custom web scraper inside apify but it doesnt retrieve data from .json output which I need
31 Replies
Is it js or python? Is it using crawlee?
adverse-sapphire•3w ago
Please move your others messages here to keep the general channel clean
adverse-sapphire•3w ago
okay
So you're saying you're able to run flawlessly from within Apify console but unable to get the output from within n8n?
I see that you're starting it from n8n correct?
adverse-sapphire•3w ago
correct
i tried an apify node as well as an http request
Are you using the official Apify node or raw HTTP calls?
Maybe you're using an asynchronous mode?
adverse-sapphire•3w ago
for now http calls seem to be better
@Patrik Barracuda Šlachta just advanced to level 1! Thanks for your contributions! 🎉
adverse-sapphire•3w ago
i tried an Apify call but it just wouldnt work
What do you meant wouldn't work? Could you please share the API endpoint you're using?
adverse-sapphire•3w ago
https://api.apify.com/v2/datasets/d4lueKKWGPt9tFfxb/items?format=json&clean=true&limit=100&desc=true + auth bearer w/ my token
and I tried asking any LLM but they just wouldnt help me at all
I was referring to the API endpoint you're doing a POST request to to see whether its sync or async
Please mention the exact thing that doesn't work.
What output do you get when you trigger the run using HTTP request?
adverse-sapphire•3w ago
adverse-sapphire•3w ago
adverse-sapphire•3w ago
so I am using an HTTP rrequest because the custom scraper isn't having an apify call
Something to pinpoint in your 2nd screenshot is that you're doing a POST request to read a dataset.
To read a dataset you'd have to send GET request. (Assuming the dataset exists).
POST should be used to trigger a run, use a synchronous endpoint to wait for it to finish. Please refer to this section https://docs.apify.com/academy/api/run-actor-and-retrieve-data-via-api#synchronous-flow, or refer your LLM to its content
Run Actor and retrieve data via API | Academy | Apify Documentation
**Learn how to run an Actor/task via the Apify API, wait for the job to finish, and retrieve its output data.
In first screenshot (using Apify node), you don't need to do anything, (HTTP calls/response are implemented, you'd just have to trigger it + waitForFinish, and you'd get your data from the default dataset associated with that run
Or is that you don't see the actor you want to trigger in the list of actors? (Referring to Apify n8n screenshot). If that's the case, please double check your authentification (Apify API key connection)
adverse-sapphire•3w ago
okay so I first post the request for it to run and then I have it inside the body as a .json code to retrieve the data from the run?
and I have the key in the header
...
is it better for me to run an HTTP Request or an Apify node?
Definitely the Apify node is the way to go, it's official and no need to handle the nitty details of HTTP requests (especially when you're not technical)
Please let me know if you get it to work
adverse-sapphire•3w ago
I'm trying to make it
I can't seem to be able to set it up
I'm trying to POST it and retrieve the data even with using a .json string accurately but it just isn't possible
we are trying to automate second half sport betting with 85% accuracy based on TA
@Patrik Barracuda Šlachta just advanced to level 2! Thanks for your contributions! 🎉
adverse-sapphire•3w ago
and we need an accurate halftime detecting tool
https://api.apify.com/v2/datasets/dA1ueKKWGPt9tFfxb/items?format=json&limit=1000&clean=true
can you help me please?
because the API from sportsdata.io would have cost us at least 10,000$ so we were looking at running queries for every single upcoming match every 6 minutes to see if it's at halftime
so if you can help me we can maybe exchange "the odds api" as well which is another 2,500$ a year
What's the ID of your actor you're trying to run?
I see you're trying to run apify/web-scraper with this input, let me check..
I've managed to run the actor you're trying to run and it's working OK
For the input, copy/paste the JSON data from here https://console.apify.com/view/runs/QS0CgK4pcnpPdKyhr to the n8n Apify node
⚠️ And don’t ever post your Apify token publicly again (Remove it asap)
adverse-sapphire•3w ago
thank you, thank you, thank you 😉
Glad it helped! If you can afford, consider hiring someone a bit technical occasionally to make your tech life easier 😃
adverse-sapphire•3w ago
I am being hired to finish this solution 😄
I own an AI Studio in San Francisco 😄
our solutions also include "Auction aggregator" which is heavily dependant on Apify
worldwide -> meaning it scrapes all auctions worldwide and, using a RAG system, find the best solution
...
3 monitors, 2 laptops (2021 year) are 60$...
AI pn built on top of API pnhub scraper
with tencent engine
huanyuan
there is a lot of solutions 😄
not sure I get what you mean, but it sounds positive 😆