wise-white•2y ago

Not sure if this is a feature request,

Not sure if this is a feature request, but it's an idea I implemented and there doesn't seem to be a better channel for general chatting. When using Scrapy, there's this concept of Items. I was able to extend Items so that they contain declarative specification of Apify input types (see code):

class ExchangeRate(Item):
    code = Field()
    rate = Field(apify_format="number")

class ExchangeRate(Item):
    code = Field()
    rate = Field(apify_format="number")

Then I go through these classes and generate output schemas for Apify datasets automatically: - CLI - schema generation Perhaps this is too specific to my project, perhaps this can inspire someone to simplify their workflow.

13 Replies

Lukas Krivka•2y ago

cc @Vlada Dusek

Vlada Dusek•2y ago

Hi, thanks Honza for the input, we were already discussing this in DMs. Unfortunately, currently, we don't have the capacity to work on adding new features in our Scrapy integration, at least in this quarter.

ratty-blush•2y ago

Yup, thanks for all the work done nevertheless!

wise-white•2y ago

I was wondering, is there any way to get user's input with scrapy cuz last time I tried it didn't end well

ratty-blush•2y ago

The Scrapy template works with actor inputs, if you mean those E.g. the proxy settings are there, but anything can be there

Vlada Dusek•2y ago

@Kirada , sure, check this part of the actor template - https://github.com/apify/actor-templates/blob/dc5e68805dcf630f35d112a7e113e4f388bbf30a/templates/python-scrapy/src/main.py#L79:L82

GitHub

actor-templates/templates/python-scrapy/src/main.py at dc5e68805dcf...

This project is the :house: home of Apify actor template projects to help users quickly get started. - apify/actor-templates

wise-white•2y ago

I ran the template and it goes on an infinite run I think it's bugged

wise-white•2y ago

wise-white•2y ago

it's been running for a few minutes even tho it finished exact same issues I had before

Vlada Dusek•2y ago

@Kirada Hi, are you using the newest version of Apify SDK (1.5.3) and the newest version of our Scrapy template (https://github.com/apify/actor-templates/tree/master/templates/python-scrapy)? If not, please do an upgrade of your Actor. I'm aware of this infinite run bug, however, I believe it should be fixed now!

wise-white•2y ago

I cloned the repo and used the template from there

wise-white•2y ago

(https://github.com/apify/actor-templates/tree/master/templates/python-scrapy

GitHub

actor-templates/templates/python-scrapy at master · apify/actor-tem...

This project is the :house: home of Apify actor template projects to help users quickly get started. - apify/actor-templates

wise-white•2y ago

I cloned this

Not sure if this is a feature request,

Did you find this page helpful?