dsy401
August 21, 2023, 1:57am
1
Hi developers,
Recently, I migrated the apollo server v3 to v4 on around 17 August and experience there are spikes on load balancer average response time.
Then I checked the server memory and it is not stable as well.
So, I am assuming whether it is related to apollo server v4 migration.
I was trying to use k6 load testing to conduct the pressure testing to simulate the high concurrency requests to hit my local server (Apollo server v4), and I have some logging that shows the memory heap usage increase rapidly.
Then I switched back to Apollo server v3, and conducted the high concurrency requests to hit my local server, and the logging shows the memory heap usage increase smoothly comparing to apollo server v4.
I also trying to switch the node version between 14, 16 and 18, the result was still the same.
I have no idea what the issue will be. Can anyone give me help?
Dependencies
"graphql": "^15.7.2", // also tried to upgrade to 16, still get the same result
"@apollo/server": "^4.9.1"
node version: 14.16
Hey @dsy401 , thanks for reporting. Can you tell me if the issue still exists when you add the ApolloServerPluginUsageReportingDisabled() plugin to your plugins configuration? This might be related to an issue that’s already open:
opened 07:26AM - 12 Jul 23 UTC
### Issue Description
We've been running Apollo server for a while in a couple … of APIs, and we have always noticed a memory leak in both, which appeared to be linearly proportional to the number of requests handled by each API.
While investigating the memory leak, v8 heap snapshots where taken from the running servers at two different timestamps, with a distance of 6 hours. The latter heap snapshot was compared to the previous one in order to track what new objects are in the JS heap that where not 6 hours before, and there are thousands of new retained `Request`-like objects that reference the "usage-reporting.api.apollographql.com" host, and hundreds of `TLSSocket` new objects that reference this same host.
### Some objects that are leaking in the JS memory:
<details>
<summary><b>Request-like object</b></summary>
```
body::Object@13534193
cache::"default"@729🗖
client::Object@13537293
credentials::"same-origin"@54437🗖
cryptoGraphicsNonceMetadata::""@77🗖
destination::""@77🗖
done::system / Oddball@73🗖
headersList::HeadersList@13537317
historyNavigation::system / Oddball@75🗖
initiator::""@77🗖
integrity::""@77🗖
keepalive::system / Oddball@75🗖
localURLsOnly::system / Oddball@75🗖
map::system / Map@130579
method::"POST"@49427🗖
mode::"cors"@84517🗖
origin::system / Oddball@67🗖
parserMetadata::""@77🗖
policyContainer::Object@13537295
preventNoCacheCacheControlHeaderModification::system / Oddball@75🗖
priority::system / Oddball@71🗖
properties::system / PropertyArray@13537319
redirect::"follow"@53093🗖
referrer::"no-referrer"@85507🗖
referrerPolicy::system / Oddball@67🗖
reloadNavigation::system / Oddball@75🗖
replacesClientId::""@77🗖
reservedClient::system / Oddball@71🗖
responseTainting::"basic"@102749🗖
serviceWorkers::"none"@519🗖
taintedOrigin::system / Oddball@75🗖
timingAllowFailed::system / Oddball@75🗖
unsafeRequest::system / Oddball@75🗖
url::URL@13537301
<symbol context>::URLContext@13538143
fragment::system / Oddball@71🗖
host::"usage-reporting.api.apollographql.com"@13538145🗖
map::system / Map@135759
path::Array@13538147
port::system / Oddball@71🗖
query::system / Oddball@71🗖
scheme::"https:"@6945🗖
username::""@77🗖
__proto__::Object@135757
<symbol query>::URLSearchParams@13538149
map::system / Map@135741
__proto__::URL@135739🗖
urlList::Array@13537299
useCORSPreflightFlag::system / Oddball@75🗖
useCredentials::system / Oddball@75🗖
userActivation::system / Oddball@75🗖
window::"no-window"@87117🗖
__proto__
```
</details>
<details>
<summary><b>TLSSocket object</b></summary>
```
<symbol blocking>::system / Oddball@75🗖
<symbol client>::Client@131765
<symbol connect-options>::Object@13536139
<symbol error>::InformationalError@13536143
<symbol kBuffer>::system / Oddball@71🗖
<symbol kBufferCb>::system / Oddball@71🗖
<symbol kBufferGen>::system / Oddball@71🗖
<symbol kCapture>::system / Oddball@75🗖
<symbol kHandle>::system / Oddball@71🗖
<symbol kSetKeepAlive>::system / Oddball@75🗖
<symbol kSetNoDelay>::system / Oddball@73🗖
<symbol maxRequestsPerClient>::system / Oddball@67🗖
<symbol no ref>::system / Oddball@73🗖
<symbol parser>::system / Oddball@71🗖
<symbol pendingSession>::system / Oddball@71🗖
<symbol res>::system / Oddball@71🗖
<symbol reset>::system / Oddball@75🗖
<symbol timeout>::system / Oddball@71🗖
<symbol verified>::system / Oddball@73🗖
<symbol writing>::system / Oddball@75🗖
_SNICallback::system / Oddball@71🗖
_closeAfterHandlingError::system / Oddball@75🗖
_controlReleased::system / Oddball@73🗖
_events::Object@13536133
_hadError::system / Oddball@75🗖
_host::"usage-reporting.api.apollographql.com"@131813🗖
_maxListeners::system / Oddball@67🗖
_newSessionPending::system / Oddball@75🗖
_parent::system / Oddball@71🗖
_peername::Object@13536141
_pendingData::system / Oddball@71🗖
_pendingEncoding::""@77🗖
_readableState::ReadableState@13536135
_rejectUnauthorized::system / Oddball@73🗖
_requestCert::system / Oddball@73🗖
_secureEstablished::system / Oddball@73🗖
_securePending::system / Oddball@75🗖
_server::system / Oddball@71🗖
_sockname::system / Oddball@71🗖
_tlsOptions::Object@13536129
_writableState::WritableState@13536137
allowHalfOpen::system / Oddball@75🗖
alpnProtocol::system / Oddball@75🗖
authorizationError::system / Oddball@71🗖
authorized::system / Oddball@73🗖
connecting::system / Oddball@75🗖
domain::system / Oddball@71🗖
encrypted::system / Oddball@73🗖
map::system / Map@130053
properties::system / PropertyArray@13536145
secureConnecting::system / Oddball@75🗖
server::system / Oddball@67🗖
servername::"usage-reporting.api.apollographql.com"@13536131🗖
ssl::system / Oddball@71🗖
__proto__::Socket@147607🗖
```
</details>
Here is a chart showing the memory usage of the last two days for one of the APIs:
The first left half of the chart (the first day) the Apollo server was running with the `ApolloServerPluginUsageReporting` enabled, and the memory kept increasing linearly, and the last half (the second day), exactly the same code was running but passing the `ApolloServerPluginUsageReportingDisabled` to the plugins, so that the usage reporting is disabled. In this last case no memory was being leaked.
We are using `@apollo/server` with version `4.3.0`
### Link to Reproduction
https://github.com/GabrielMusatMestre/apollo-server-memory-leak-repro
### Reproduction Steps
Steps are described in the README.md of the reproduction repo.
This is not a reliable reproduction, as the memory leak might start being noticeable by running the server under heavy load for hours or days, and it needs a properly configured `APOLLO_KEY` and `APOLLO_GRAPH_REF` that will actually publish usage reports to Apollo.
If it seems unrelated, would you please open another issue on the apollo-server repo with as much detail as you have available and a reproduction? Without more details I’m not sure I can reproduce this, so please be as thorough as possible.
dsy401
August 21, 2023, 10:03pm
3
Thanks, I will add the ApolloServerPluginUsageReportingDisabled() to see whether it works or not
dsy401
August 22, 2023, 1:33am
4
@trevor.scheer Looks like there were still memory leak and performance issue. I will create issue in apollo-server repo
@dsy401 did you create the issue? We are also having a memory leak but we cannot find it yet.
