[BUG] K8s Opensearch dashboard reporting download error

[shcohen2002]

What is the bug?
When downloading a report as a PDF or PNG the file doesn't download.
I get a popup message "Download Error, There was an error generating this report" and an error '504 Gateway Time-out' in the network analyzer.

I have followed these links but there wasn't any success finding a solution:
opensearch-project/reporting#575
https://forum.opensearch.org/t/opensearch-dashboard-reporting-download-error/11281
https://forum.opensearch.org/t/kibana-reporting-getting-error-while-downloading-reports/4414

I am installing opensearch using helm on my k8s cluster.
When experimenting opensearch on my laptop using docker-compose, the report is generated and downloaded.
So i guess it's a miss configuration or a bug.
https://opensearch.org/docs/1.3/opensearch/install/docker/

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to 'reports-dashboards'
2. Click on 'PNG or PDF' to download the report, then
   Loader popup presented.
3. See error - "Download Error, There was an error generating this report"

OR

1. Go to 'dashboards' and click on a dashboard.
2. Click on 'Reporting' from the upper right side.
3. On the opened menu click 'Download PDF' or 'Download PNG'
   Loader popup presented.
4. See error - "Download Error, There was an error generating this report"

What is the expected behavior?
After clicking on 'PNG or PDF' the report is downloaded.

OpenSearch Version
CHART VERSION:
opensearch-2.8.0
APP VERSION
6.4.0

Dashboards Version
CHART VERSION:
opensearch-dashboards-2.6.0
APP VERSION
2.4.0

What is your host/environment?
kubernetes 1.23 configured with 3 nodes

Do you have any screenshots?

Do you have any additional context?
opensearch_dashboards.yml:

   # Use HTTPS instead of HTTP
   opensearch.hosts: [https://opensearch-cluster-master:9200]
   # Configure the OpenSearch Dashboards internal server user
   #opensearch.username: kibanaserver
   #opensearch.password: *************
   # Disable SSL verification when using self-signed demo certificates
   opensearch.ssl.verificationMode: none
   # allowlist basic headers and multi-tenancy header
   opensearch.requestHeadersAllowlist: [authorization, securitytenant]

   opensearch_security.multitenancy.enabled: true
   opensearch_security.multitenancy.tenants.preferred: [Private, Global]
   opensearch_security.readonly_mode.roles: [kibana_read_only]
   # Use this setting if you are running opensearch-dashboards without https
   opensearch_security.cookie.secure: false
   server.host: '0.0.0.0'
   #opensearch_reporting.osd_server.hostname: '127.0.0.1'

   opensearch.ssl.certificate: ***
   opensearch.ssl.key: ***
   opensearch.ssl.certificateAuthorities: ***

Node logs:
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
[2022-12-17T10:54:15,533][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-12-17T10:54:15,576][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-1] loaded plugin [opensearch-reports-scheduler]
[2022-12-17T10:54:25,377][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-1] JobSweeper started listening to operations on index .opendistro-reports-definitions
[2022-12-17T10:54:37,285][INFO ][o.o.i.s.IndexShard ] [opensearch-cluster-master-1] [.opendistro-reports-instances][0] detected new primary with primary term [47], global checkpoint [7], max_seq_no [7]
[2022-12-17T10:54:37,452][INFO ][o.o.i.s.IndexShard ] [opensearch-cluster-master-1] [.opendistro-reports-definitions][0] detected new primary with primary term [8], global checkpoint [-1], max_seq_no [-1]
[2022-12-17T10:57:16,768][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-createOnDemand
[2022-12-17T10:58:00,789][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-getAll fromIndex:0 maxItems:10000
[2022-12-17T10:58:01,034][INFO ][o.o.r.i.ReportInstancesIndex] [opensearch-cluster-master-1] reports:getAllReportInstances from:0, maxItems:10000, retCount:9, totalCount:9
[2022-12-17T10:58:02,489][INFO ][o.o.r.a.ReportDefinitionActions] [opensearch-cluster-master-1] reports:ReportDefinition-getAll fromIndex:0 maxItems:10000
[2022-12-17T10:58:02,503][INFO ][o.o.r.i.ReportDefinitionsIndex] [opensearch-cluster-master-1] reports:getAllReportDefinitions from:0, maxItems:10000, retCount:0, totalCount:0
[2022-12-17T11:14:39,388][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-getAll fromIndex:0 maxItems:10000
[2022-12-17T11:14:39,394][INFO ][o.o.r.a.ReportDefinitionActions] [opensearch-cluster-master-1] reports:ReportDefinition-getAll fromIndex:0 maxItems:10000
[2022-12-17T11:14:39,401][INFO ][o.o.r.i.ReportDefinitionsIndex] [opensearch-cluster-master-1] reports:getAllReportDefinitions from:0, maxItems:10000, retCount:0, totalCount:0
[2022-12-17T11:14:39,404][INFO ][o.o.r.i.ReportInstancesIndex] [opensearch-cluster-master-1] reports:getAllReportInstances from:0, maxItems:10000, retCount:9, totalCount:9
[2022-12-17T11:14:48,116][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-info iP27H4UBEcFBm0zOJZDk
[2022-12-17T11:57:56,290][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-createOnDemand
[2022-12-17T11:59:16,088][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-createOnDemand
[2022-12-17T12:06:38,284][INFO ][o.o.r.a.ReportInstanceActions] [opensearch-cluster-master-1] reports:ReportInstance-createOnDemand
[2022-12-17T12:42:08,385][INFO ][o.o.r.a.ReportDefinitionActions] [opensearch-cluster-master-1] reports:ReportDefinition-getAll fromIndex:0 maxItems:10000
[2022-12-17T12:42:08,389][INFO ][o.o.r.i.ReportDefinitionsIndex] [opensearch-cluster-master-1] reports:getAllReportDefinitions from:0, maxItems:10000, retCount:0, totalCount:0

LOGS
Browser Network analyzer:
ip.linodeusercontent.com.har.zip
Dashboard:
opensearch-dashboards.logs.zip

[ToniIltanen]

I am getting the same error, the report generation ends with a timeout.

Opensearch V.2.4.0 in use

[Martin-Kemp]

Also getting a timeout with V2.4.0 to V2.6.0. Also running in K8s.

[cvestre-cisco]

Seeing the same behavior in v2.8.0.

The following API call is returning a 404:
https://[REDACTED]/api/reporting/generateReport?timezone=America%2FChicago&dateFormat=MMM+D%2C+YYYY+%40+HH%3Amm%3Ass.SSS&csvSeparator=%2C

[mmguero]

Perhaps the same as #496?

[mmguero]

Any chance of getting this looked at?

[mmguero]

still broken in v2.11.1

[krisfreedain]

@anirudha - can you have someone look at this?

Catch All Triage - 1, 2, 3