Advice on how to use combined remote data

General
1

This is my first time using Apollo Client so I’m still getting familiar with things. I need to make a combined query which fetches two types of data: Posts and Adverts. Both of these are arrays of data and once I have the data fetched from the remote source I need to merge the two arrays at a specific frequency (a separate JS function). This will result in a single array of data which is to be used as the content for my page in a Next.js application.

My question is how is it best to handle that combined array of data? Should I store it in React state, use Apollo Reactive Variables, or write to cache?

To provide a little extra info on the usage:

  • This combined data will be passed to a couple of other components which need to re-render when the content changes.
  • I need data to be fetched automatically on initial page load, after that it will be triggered based on user interaction (interacting with a search form and clicking Submit, or use pagination controls to page through the data)
  • Every time a user submits a new search I need to make sure I fetch the remote data according to the search criteria they’ve entered so this will alter the Post data that is returned and a new combined array needs to be created accordingly.

Any advice is greatly appreciated.

2

Hey @ian :wave:

I’ll do my best to describe what I’d do in your situation and try and give the reasons why. Just a disclaimer, this is my personal opinion on how to structure this so this is no means the only “right” way to do it.

I need to merge the two arrays at a specific frequency (a separate JS function). This will result in a single array of data which is to be used as the content for my page

If I’m reading this correctly you’re combining these arrays is because you’d like to display a single list in your UI with the set of combined data correct? You mentioned that a user will submit some kind of search, so I’m guessing this is some kind of search page?

My question is how is it best to handle that combined array of data? Should I store it in React state, use Apollo Reactive Variables, or write to cache?

Let me mention write off the bat that your data will automatically be cached for you in the Apollo cache, unless you decide to change your fetchPolicy to no-cache. Because of this, you shouldn’t need to do too much on your own to store the data manually. I’d recommend installing the Apollo Client Devtools if you don’t already have them, which will let you explore the cache and how that data is stored after it is fetched. This page gives a good overview on how data is cached inside Apollo’s InMemoryCache so I’d recommend giving that a look as well.

Assuming you’re using one of the query hooks (useQuery, useSuspenseQuery, useLazyQuery, etc.) in your app, these hooks will automatically rerender your component for you anytime data in the cache changes for your query. This means that a user interaction that would refetch that query should automatically re-render your component with the updated data.

Since the hooks handle rerendering your components when data changes in the cache, I would not recommend trying to copy that data anywhere else, otherwise you’re going to have to manually sync the data to your other data store anytime it changes.

Back to the original question. Given your requirements, I tend to view this type of UI as a rendering concern rather than a data concern. This means I would avoid trying to do this type of data manipulation at the cache level and instead combine this data in the components that render this data.

My reason for this is because you have no idea what your requirements are down the line, and its very possible you may end up needing to “reuse” data from one of those arrays in another area of your UI. Apollo tries to use data in the cache as much as possible before it goes to the network, which means that multiple queries could depend on overlapping data that may already be stored in the cache. If you end up manipulating this data at the cache level, this might make it harder down the line to try and reuse that data for other areas of your UI since you’ve modified it to work for a single screen (hopefully this makes sense).

Again, I view this as a rendering concern since the combined data set is what you want to display. I’d just merge the data together in those components.

const SEARCH_QUERY = gql`
  query SearchQuery {
    # ...
  }
`
function Search() {
  const { data, loading } = useQuery(SEARCH_QUERY, { variables })

  const combinedItems = combineDataSets(data.array1, data.array2)

  return <SearchData items={combinedItems} />
}

Again the advantage here is that Apollo will automatically re-render your components when data changes, which means you don’t need to sync that data to any other “store” in your app.

Hopefully this makes sense and hope this gives you a good direction to start!

1 Like
3

Hey @jerelmiller! Thanks for taking the time to write such a helpful and informative post.

Your assumptions are correct, I’m combining the arrays to display a single list in the UI with the combined data, and yes, it’s a search page with select fields providing filters to that search.

Thanks for the information and link on Apollo cache. That helped me to build a better picture of Apollo caching.

I’ve tried installing Apollo Client Devtools but when developing on localhost I can’t get the Apollo tab to display in devtools. Does it not work on localhost? It does seem to display when viewing my branch deployment…(?)

Since the hooks handle rerendering your components when data changes in the cache, I would not recommend trying to copy that data anywhere else, otherwise you’re going to have to manually sync the data to your other data store anytime it changes.

Really helpful tip. Exactly what I was concerned with if I kept a local state of the combined properties.

Given your requirements, I tend to view this type of UI as a rendering concern rather than a data concern. This means I would avoid trying to do this type of data manipulation at the cache level and instead combine this data in the components that render this data.

A good way to frame it. That makes complete sense and from giving it a try I see that it works perfectly!

Thanks again for the really helpful information. Much appreciated.

4

Although I’ve got the combined array assembled on the client-side, can you suggest a way I can incorporate pagination? It’s not something I’m going to be able to request directly from the API with limit and offset.

5

Glad this is helpful for you! Let us know if you have any other questions!

Thats odd, it should work just as well locally as it does in prod. Can you try configuring your bundler as shown in this doc? We use a global __DEV__ variable to determine whether an app is in development or not, so its possible Apollo Client is not able to infer this correctly based on how your bundler is setup.

If this doesn’t work, can you try setting connectToDevTools: true when you create the ApolloClient instance to force it to try and connect?

const client = new ApolloClient({
  connectToDevTools: true,
  // ...other options
})

Hopefully one of these two methods work for getting it to connect for you.

can you suggest a way I can incorporate pagination?

This is going to be a bit trickier since you’re trying to paginate on two different arrays. Out of curiosity, do you have access or control on the backend schema at all? If so, you might consider introducing a search field with a union type that lets you return multiple types on a single field and also allows you to handle pagination on the backend.

If you don’t have this level of control, this gets a bit trickier since its difficult to determine what is the “next page” between the two data sets. Out of curiosity, is your pagination implemented using offset-based, cursor-based pagination, or some other way of doing it? Also out of curiosity, what are you using to determine how to merge those two arrays?

6

Regarding Apollo Devtools, I’m using Next.js so I’m not directly configuring the bundler, just going largely with the out-of-the-box defaults. I have had success by forcing the issue with connectToDevTools: true, though. That finally sees the Apollo tab appear in the browser. Is this necessary all the time with Next.js?

Regarding pagination, it’s a yes and no to having control of the backend schema. Yes, in that I’m providing the backend so can make modifications, however, I’m using MongoDB Atlas App Services to provide the GraphQL API and currently they don’t support the Union type. :roll_eyes:

They also don’t support pagination out-of-the-box but I have built in an offset param along with a limit providing me with offset-based pagination. This works well if I only need to request one type but since the types are not combined it becomes fairly useless in this situation. :confused:

7

In terms of how I’m merging the two arrays, here’s my function in action. Essentially it just takes the two arrays as input along with start and frequency params and then intersplices the two accordingly. I do shuffle one of the arrays first as well.

I can only think that I would need to memoize the function that merges the two arrays and then array.slice() in line with the page size?

(I had to split this reply in two as I could only add two links per post)

8

By way of an update, I’ve managed to get things working on the client-side by requested both data sets separately and then combining into a single array. I’ve then used useMemo to memoize that array so that I can refer to that one for paging.

After that I use the URL (page query param) to determine which page is currently being viewed and then slice the array up according to pageSize and page number. That’s then fed to the UI to display. I’ve got it working with pagination controls where I update the component state after checking the URL.

The downsides are that I have to request both data sets in full all at once and then mix them at runtime into an even larger array with processing involved. For every page change I also then have to reprocess that array to know which slice to show. It works on desktop and the delay isn’t terrible but quicker is always better. On mobile it seems to error for whatever reason. I’m not sure if this is due to too much memory or processing being demanded? The arrays aren’t huge, currently under 1000 items but this can will like likely increase a little to maybe up to 2000 max.

I also notice that on every page request I still see the graphql requests going out over the network. Does this mean Apollo Client isn’t caching anything and always going out over the network? How do I know what’s cached and what isn’t?

I’m still open to suggestions however, whether it be for improvements on the client-side or if there is an alternative viable server-side solution which would be preferable. I’m looking to host on Vercel so I do have access to serverless functions which technically could run a GraphQL server instead of using the one from MongoDB Atlas. I may be able to combine the two data sets into a single response server-side then using Union. However I do plan on using MongoDB Atlas for user authentication later on just as a consideration. Also taking into consideration the extra work it would take to setup and manage a server-side graphql server component via Next.js and serverless. I’ve not done that before to really know the extra effort involved.

I’m open to suggestions though with all things considered.

Thanks.

9

Hey @ian, apologies for my delay in reply! I promise I’ll come back and give you a proper reply as soon as I’ve got a bit of time! Just wanted to let you know I hadn’t forgotten about this thread :slightly_smiling_face:

10

Hey @jerelmiller. Thanks for the update, and no worries. I appreciate the advice so I’m happy to wait until you get a moment. :slightly_smiling_face:

In the meantime I’ve looked into Apollo Server to investigate the feasibility. I’ve not set up a GraphQL server before but it looks like Apollo Server v4 has a useful integration with Next.js. In theory it doesn’t seem too tricky to try and setup a server component for my needs. I’ve not used Union before either but seems like it should allow me to do what I need. I’d have some questions about how to do the shuffle and combine in the resolver but otherwise, in theory, it seems like it could be a viable option to setup the GraphQL server and remove the burden from the client. I would probably need to protect the route with authentication however.

I’m sure putting it into practice would present it’s own difficulties though… :thinking:

11

Handling a combined array of data in a React application using Apollo Client involves several considerations. Here’s a suggested approach:

1. Apollo Client Queries:

Define GraphQL queries for fetching Posts and Adverts separately. You can use the useQuery hook from @apollo/client to fetch data.

const { loading: postsLoading, data: postsData } = useQuery(GET_POSTS);
const { loading: advertsLoading, data: advertsData } = useQuery(GET_ADVERTS);

const { loading: postsLoading, data: postsData } = useQuery(GET_POSTS);
const { loading: advertsLoading, data: advertsData } = useQuery(GET_ADVERTS);

2. Combining Data:

Once both queries are completed, you can merge the data using a separate JavaScript function. You might want to use a function like:

const mergeData = (posts, adverts) => {
// Your merging logic here
};

const mergeData = (posts, adverts) => {
  // Your merging logic here
};

3. Handling State:

Now, to handle the combined data in your component, you can use React state. You can use the useState hook to manage the state of the combined array:

const [combinedData, setCombinedData] = useState();

const [combinedData, setCombinedData] = useState([]);

4. Updating State:

Whenever the posts or adverts data changes, call your merging function and update the state:

useEffect(() => {
if (!postsLoading && !advertsLoading) {
const mergedData = mergeData(postsData.posts, advertsData.adverts);
setCombinedData(mergedData);
}
}, [postsData, advertsData]);

useEffect(() => {
  if (!postsLoading && !advertsLoading) {
    const mergedData = mergeData(postsData.posts, advertsData.adverts);
    setCombinedData(mergedData);
  }
}, [postsData, advertsData]);

5. Fetching Data:

For automatic fetching on initial load or user interaction (like search form submission), you can use Apollo Client’s refetch method or trigger the queries manually.

6. Passing Data to Components:

Pass the combinedData to the components that need it, and they will re-render when the content changes.

7. Reactive Variables or Cache:

Reactive variables or writing directly to the cache might be useful in more complex scenarios. However, for your use case, managing the combined data in local state should suffice.

In summary, using local state (useState) to manage the combined data, along with Apollo Client for querying and refetching, should provide a clean and effective solution for your use case in a Next.js application.

Celvin Klassen
Business Consultant

12

Hi @KlevinKlassen, thanks for the excellent summary. That was almost exactly what I was doing when trying to combine things on the client-side. It worked but on reflection it felt like adding the extra processing on the client-side wasn’t the best approach. Luckily in my situation I was able to implement Apollo Server and do the combining and mixing server-side before returning to the client.

The only thing I’ve not being able to get working is the client-side caching in Apollo Client. This mainly seems to be due to the less straight-forward case of having a union of types returned. So far I think I’ve been able to get the writing to cache store working but not reading from it. It’s something I need to revisit when I get time.

Thanks for your help.

13

@ian have you configured possibleTypes for the cache? When using union/interface types, you’ll want to declare these to ensure the cache reads this data properly. Hope this helps!

14

@jerelmiller no, I haven’t configured possibleTypes yet. Maybe that’s what I’m missing. Thanks for the tip, I’ll give it a try.