All "elastically scalable / autoscaling" cloud services (e.g. S3, ELB/ALB, Aurora Serverless, etc.) are ultimately composed of application code running on a finite number of machines thereby capable of serving a finite throughput of load. Excess load must be temporarily shed while additional machines are spun up. You can easily observe this behavior by running load tests on these services.

Separately, all applications don't scale linearly with the volume of open connections. There's typically a sweet spot of open connections that provides maximum throughput. Exceeding this sweet spot will actually reduce aggregate throughput. To the extent that serverless applications are stateless and are unable to pool connections as well as a stateful application, you should expect to see the volume of connections to grow proportional to load, potentially knocking over the backing DB (or in Aurora Stateless's case, returning max connections errors).