Store json response as file (in s3 bucket)

Help
1

my graphql response payload is a very huge json object(1-80MB). So Instead of returning the json as response I am planning to store it in file. I m doing this in formatResponse of ApolloServer.

For small size file it is working but for big file it don’t. I am assuming this is because s3.upload is time taking process but fomatResponse finishes before uploading.

also formatResponse does not support async wait. so I cannot make code to wait the upload to finish.
I m not sure what is the best place to perform s3 upload to save my response as file.

2

Can you describe what kind of data we’re talking about that would be 80MB?

As in, why would it go from 1-80MB?

3

Hi Kevin, I am using graphql to fetch marketstock data. So based on what data range is selected data will vary in size. and this data will be json.

I have solved this problem as of now using server.createHandler()(event, context, callbackFilter); where callbackFilter fn can be used to see response and work on it . and this fn can use aysnc await which solve my problem.

any efficient solution is welcomed.

4

Sounds like you may benefit from caching the results and using Relay-style pagination to allow you to more efficiently return results. Depending on your downstream needs, this may have the benefit of allowing clients or data pipelines to have smaller amounts of data to work on, allowing you to parallelize or only look at a certain slice of data at a time.

1 Like