Parameters

https://valiantys.com/en/blog/atlassian-administration/indexing-jira/

Before test

A background index took 20 hours.

CPUs: 8

Memory: 24 GB (16 for Jira)

JAVA_OPTS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Djava.util.logging.config.file=/pack/jira/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/javax.crypto=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.base/sun.reflect.generics.parser=ALL-UNNAMED --add-opens=java.base/java.time=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED --add-exports=java.base/sun.util.calendar=ALL-UNNAMED --add-exports=java.base/sun.security.action=ALL-UNNAMED --add-exports=java.xml/jdk.xml.internal=ALL-UNNAMED -Xms15360m -Xmx15360m -XX:InitialCodeCacheSize=32m -XX:ReservedCodeCacheSize=512m -Dlog4j2.contextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dlog4j2.disableJmx=true -Dlog4j2.garbagefree.threadContextMap=true -Dlog4j2.isWebapp=false -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.dom4j.factory=com.atlassian.core.xml.InterningDocumentFactory -Datlassian.mail.senddisabled=true -Datlassian.mail.fetchdisabled=true -Datlassian.mail.popdisabled=true -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -DdisableLicenseForStatuspageIntegration=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/pack/jira-heap-dumps -XX:-OmitStackTraceInFastThrow -Djava.locale.providers=COMPAT -Datlassian.plugins.startup.options= -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xlog:gc*:file=/pack/jira/logs/atlassian-jira-gc-%t.log:tags,time,uptime,level:filecount=5,filesize=20M -XX:+ExplicitGCInvokesConcurrent -Dignore.endorsed.dirs= -Dcatalina.base=/pack/jira -Dcatalina.home=/pack/jira -Djava.io.tmpdir=/pack/jira/temp

Caveats:

The clone has no users, no jobs, no mail - hence no load.

Looking at the index times in prod and on the clone, the dirrefence is way smaller (if any) from prodto clone, so a load on the prod cant explain the slowlyness.

Test: Background Reindex directly after cloning

For at se om der er samme lousy performace

Background Reindex started 08:54

3% at 09:17 (23 min) → 8min/% →  800 minutes estimate

5% at 09:35 (41 min) → 8min/% → 800 minutes estimate

Stopping, as the klon has proven significan "slowness" as in production.

The difference from real Prod (with an actual 1200 minute time) is problably the load/usage difference

Test: Full Reindex directly after cloning

The clone is rolled back to snapshot

Stop Jira

Delete index on disk (rm -rf caches/)

Start Jira

Full reindex started at 10:20

10% at 10:27 (6,5 min) → 1,5min/% → 150 minutes estimate

20% at 10:33 (12 min) → 1,6min/% → 160 minutes estimate

Stopping. Full Reindex is for sure faster, but stil not as expected.

Test: More threads:

The clone is rolled back to snapshot

Stop Jira

Increase threads as stated in https://confluence.atlassian.com/jirakb/how-to-increase-the-speed-of-full-reindex-in-jira-826879636.html

jira.index.issue.threads = 30
jira.index.sharedentity.threads = 30

added to file jira-config.properties

Start Jira

Background Reindex started 10:45

2% at 10:59 (14 min) → 7min/% → 700 minutes estimate

Stopping, this was like the original prod.

Test: Uninstall scriptrunner to test without scripted fields

Scripted fields does have an impact, as Groovy are parses og executed. If a field uses the search or references another scripted field, this could cause serious problems.

The clone is rolled back to snapshot

Scriptrunner is uninstalled

Background Reindex started 09:42

3% at 10:03 (21 minutes) → 7min/% → 700 minutes estimate

5% at 10:12 (30 min) → 6min/% → 600 minute estimate

Stopping, as there has been no improvement at all.

Test: Uninstall Elements Connect

The clone is rolled back to snapshot

Elements Connect is uninstalled

Background Reindex started 13:33

Stopping, as there has been no improvement at all.

Test: Vaccum database

The clone is rolled back to snapshot

Stop Jira

Run in PSQL:

VACUUM(FULL, ANALYZE, VERBOSE);

Start Jira

Background Reindex started 12:38

1% 12:45 (7 minutes)

Stopping, as there has been no improvement at all.

Test: Increase to 16 CPU

The clone is rolled back to snapshot

Shutdown server

Increase CPUs from 8 to 16

Start server and Jira

Background Reindex started 11:13

3% at 11:35 (22min) → 7min/% -> 700 minutes estimate

5% at 11:55 (42 min) → 8min/% → 800 minutes estimate

Stopping, as there has been no improvement at all.

Test: Increase to 48 GB Ram

The clone is rolled back to snapshot

Shutdown server

Increase Ram to 48 GB (xmx=30720)

Start server and Jira

Background Reindex started 20:35

Test: Reindex in Safe mode

The clone is rolled back to snapshot

Set Jira in safe mode via GUI

Background Reindex started 13:02

4% at 13:16 (14 minutes) → 3,5min/% → 350 minutes estimate

7% at 13:24 (22 minutes) → 3min/% → 300 minutes estimate

Stopping, as there has been no improvement at all.

Test:  JIRA re-indexing fails with Error getting the next result (This ResultSet is closed.)

https://community.atlassian.com/t5/Jira-Software-questions/Increase-reindex-speed-jira-index-issue-threads-not-showing-in/qaq-p/2241037

https://confluence.atlassian.com/jirakb/jira-re-indexing-fails-with-error-getting-the-next-result-this-resultset-is-closed-346947669.html

The clone is rolled back to snapshot

Stop Jira

jira.index.issue.maxqueuesize=4000
jira.index.sharedentity.maxqueuesize=4000

Pool-max-size is already 80

Start Jira

Background Reindex started 13:33

1% at 12:42 (8 min) → 8min/% → 800 minutes estimate

Stopping, as there has been no improvement at all.

TEST: Delete attachments before reindex

The clone is rolled back to snapshot

Stop Jira

Delete attachments in data (rm -rf attachments)

Start Jira

Background Reindex started 15:18

1% at 15:25 (7 min) → 7min/% → 700 minutes estimate

Test: Postgres conf changes

The clone is rolled back to snapshot

Stop Jira

postgresql.conf is changed:

root@docs02:/etc/postgresql/14/main# diff postgresql.conf postgresql.conf.npn 
126,127d125
< #
< dynamic_shared_memory_type = posix
165c163
< #wal_level = replica			# minimal, archive, or hot_standby
---
> wal_level = replica			# minimal, archive, or hot_standby
177c175
< #full_page_writes = on			# recover from partial page writes
---
> full_page_writes = on			# recover from partial page writes
190c188
< #checkpoint_completion_target = 0.9
---
> checkpoint_completion_target = 0.9
192,193c190,191
< max_wal_size = 1GB
< min_wal_size = 80MB
---
> max_wal_size = 4GB
> min_wal_size = 1GB
root@docs02:/etc/postgresql/14/main# 

Restart server

Background Reindex started 19:54

1% at 20:04

Stopped it later, no improvements

Test: jira.cfv.driven.indexing.disabled=true 

https://confluence.atlassian.com/enterprise/optimizing-custom-fields-1005343684.html

The clone is rolled back to snapshot

jira.cfv.driven.indexing.disabled=true 

Start Jira

Background Reindex started 11:18

3% at 11:48 (30 min) → 10 min/% - 1000 minutes estimate

Stopping, as there has been no improvement at all.

Test: jira.local.context.indexing.disabled=true

https://confluence.atlassian.com/enterprise/optimizing-custom-fields-1005343684.html

The clone is rolled back to snapshot

jira.local.context.indexing.disabled=true

Start Jira

Background Reindex started 11:58

2% at 12:18 (20 min) → 10 min/% - 1000 minutes estimate

Stopping, as there has been no improvement at all.

TEST: Missing Index - JRASERVER-38598

https://jira.atlassian.com/browse/JRASERVER-38598

The clone is rolled back to snapshot

postgres=# \c jira;
You are now connected to database "jira" as user "postgres".
jira=# CREATE INDEX CONCURRENTLY idx_issue_id ON moved_issue_key (issue_id);

Start Jira

Background Reindex started 13:07

1% at 13:15 → 8min/%

Stopping, as there has been no improvement at all.

Test: Missing Index - Customfields

https://jira.atlassian.com/browse/JRASERVER-38598

The clone is rolled back to snapshot

postgres=# \c jira
You are now connected to database "jira" as user "postgres".
jira=# create index idx_netic_customfield on customfieldvalue (customfield);
CREATE INDEX
jira=# \q

Background Reindex started 09:01
1% 10 min

2% 19 min

TEST: Disk Speed

Test Instance on Fast Storage:

TOTALS
----	----	----	----	----
stat	avg	median	min	max
----	----	----	----	----
open	77,581	63,649	29,970	3,852,387
r/w	50,339	44,585	24,717	316,925
close	47,438	42,527	23,729	232,508
delete	3,157	2,643	1,771	53,527
----	----	----	----	----
All times are in nanoseconds.
root@docs02:/pack/jira-data# 

Test Instance - Standard Storage:

TOTALS
----	----	----	----	----
stat	avg	median	min	max
----	----	----	----	----
open	212,426	102,273	34,011	10,787,925
r/w	120,548	65,447	26,104	11,182,308
close	185,517	60,635	24,043	58,844,839
delete	4,037	3,312	1,648	41,369
----	----	----	----	----
All times are in nanoseconds.
root@docs02:~# 

Prod:

TOTALS
----	----	----	----	----
stat	avg	median	min	max
----	----	----	----	----
open	283,197	217,224	142,270	3,443,233
r/w	209,608	148,299	99,537	14,991,509
close	181,996	142,584	98,883	1,275,311
delete	12,966	10,342	8,765	53,228
----	----	----	----	----