limit exceeded (limit: 8388608 bytes/sec) while attempting to ingest '2140' lines totaling '3285284' bytes 429 Too Many Requests Ingestion rate limit exceeded' or '500 Internal Server Error rpc error: code = ResourceExhausted desc = grpc: received message...
="POST /loki/api/v1/push (500) 249.833406ms Response: \"rpc error: code = ResourceExhausted desc = grpc: received message larger than max (5229930 vs. 4194304)\\n\" ws: false; Accept: */*; Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3; Content-Length: 5537209; Content...
"rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4567615 vs. 4194304)" This is somthing we can configure within the server block of the config as described here https://grafana.com/docs/loki/latest/configuration/#server but unfortunately, in the loki-simp...
If Loki Canary reports a high number ofunexpected_entries, Loki Canary may not be waiting long enough and the value for the-waitflag should be increased to a larger value than 60s. Be awareof the relationship betweenpruneintervaland theinterval. For example, with an interval of 10ms (100 ...
To query log data, this index needs to be loaded, and in order to improve performance, it should probably be in memory. This is difficult to scale, and as you consume more logs, the index will quickly grow larger. Loki's index is usually an order of magnitude smaller than the amount ...
2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a --limit= far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the --batch= parameter (which defaults to 1000) until the ...
To do the hash lookup, distributors find the smallest appropriate token whose value is larger than the hash of the stream. When the replication factor is larger than 1, the next subsequent tokens (clockwise in the ring) that belong to different ingesters will also be included in the result...
If Loki Canary reports a high number ofunexpected_entries, Loki Canary may not be waiting long enough and the value for the-waitflag should be increased to a larger value than 60s. Be awareof the relationship betweenpruneintervaland theinterval. For example, with an interval of 10ms (100 ...
2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a --limit= far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the --batch= parameter (which defaults to 1000) until the ...
ResourceExhausted desc = grpc: received message larger than max #2271 Environment: Infrastructure: Loki 2.4.1, Grafana 8.5.5 👍 2 Contributor dannykopping commented Jul 4, 2022 @hterik try increasing Loki's grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size and see if ...